Turn off color for downstairs dump Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.24028757 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.05092357 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.04633419 Region set with 3 regions was created /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1187 1188 1189 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.653144721Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.653729003Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.653754321Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191} {"msg":"Crucible 129e668a-39dd-4479-b1a1-ebeaba4dc63b has session id: ae0153e3-1291-47fc-a3f3-fcf8e005edc4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.654317125Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Upstairs opts: Upstairs UUID: 129e668a-39dd-4479-b1a1-ebeaba4dc63b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.65446774Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Crucible stats registered with UUID: 129e668a-39dd-4479-b1a1-ebeaba4dc63b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.654505296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} Crucible runtime is spawned {"msg":"129e668a-39dd-4479-b1a1-ebeaba4dc63b active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.655690858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.655724135Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.655743593Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.655765841Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.655787869Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.655810807Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.655832404Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.655859352Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.65588257Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.656171551Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.656210927Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:22:55.656349193Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:22:55.65638548Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:22:55.656410657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:22:55.656435035Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:22:55.656458713Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.656477321Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.656497149Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:22:55.656519816Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:22:55.656543074Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:22:55.656569722Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.656588649Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.656607778Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:22:55.656629856Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.656648734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.656668332Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.65668783Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.656709248Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:22:55.656730295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.658428633Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.658504666Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.658539142Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_connection connected from Ok(127.0.0.1:64383)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.658726084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_connection connected from Ok(127.0.0.1:46409)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.65876508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_connection connected from Ok(127.0.0.1:63767)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.658794897Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"io task","client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.660675761Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.660703918Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.660719256Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.660733745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.660747324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.660763712Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.661762883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.6617883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.661803769Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.66249435Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662521238Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662536146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662549005Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662563064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662576502Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662589491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662606089Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662620068Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662633897Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662648445Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662662204Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662675892Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662690181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.66270377Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662718268Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:23:05.662732597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662749285Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662762824Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662775872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662789151Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"":"downstairs","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.6628043Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662817788Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"0","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662831267Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"1","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662844666Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"client":"2","":"downstairs","session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662858114Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662870683Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"129e668a-39dd-4479-b1a1-ebeaba4dc63b is now active with session: ae0153e3-1291-47fc-a3f3-fcf8e005edc4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662883632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662895631Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191,"session_id":"ae0153e3-1291-47fc-a3f3-fcf8e005edc4"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:05.662921468Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Fill test Read and Verify all blocks (0..600 range:false) Wrote out file "/tmp/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 6 Acked 1005 Read 256 Done Done Done false 7 Acked 1006 Read 88 Done Done Done false 8 Acked 1007 Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1004 1004 1004 Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 8 7 6 5 3 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1008 1007 1006 1005 1004 Upstairs last five completed: 9 8 7 6 5 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:09.747808763Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1191} Begin loop 1 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.777376692Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.777705749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.777737976Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193} {"msg":"Crucible df75da68-47b6-4525-b97e-20c9b21450db has session id: f104deb6-5051-40f3-963b-c479f51ce3f4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778046996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Upstairs opts: Upstairs UUID: df75da68-47b6-4525-b97e-20c9b21450db, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778091601Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Crucible stats registered with UUID: df75da68-47b6-4525-b97e-20c9b21450db","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778116549Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778498331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193} {"msg":"df75da68-47b6-4525-b97e-20c9b21450db active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778529288Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778553995Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778580223Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.77860382Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"1","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778626868Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"1","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778652285Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"2","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778678643Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"2","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778701301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778724978Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"io task","client":"1","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778754855Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"ds_connection connected from Ok(127.0.0.1:59060)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.778984572Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"ds_connection connected from Ok(127.0.0.1:62788)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.779297892Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"io task","client":"1","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:09.779373014Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:09.779401871Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:09.779418989Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.779434348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.779458616Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.779626669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.779827519Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"2","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.779849287Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"2","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.779863296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"1","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.779877284Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"1","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.780490133Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"2","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.78052064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"1","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.78102956Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"2","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.781056637Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"2","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.781071426Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.781084104Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"1","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.781097983Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"1","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:09.781111272Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.781306127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"ds_connection connected from Ok(127.0.0.1:39232)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.781456072Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.782341904Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.782372531Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.782931495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783529916Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783552314Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783567512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783585241Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783600369Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783614358Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783628006Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783642235Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783655914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783669562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783683611Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.78369667Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783710068Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Next flush: 3","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2024-07-09T21:23:19.783727307Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783740665Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"":"downstairs","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783754064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783766103Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"0","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783779502Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"1","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.7837946Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"client":"2","":"downstairs","session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783808479Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783820907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"df75da68-47b6-4525-b97e-20c9b21450db is now active with session: f104deb6-5051-40f3-963b-c479f51ce3f4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783833966Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783846105Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193,"session_id":"f104deb6-5051-40f3-963b-c479f51ce3f4"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:19.783858344Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 459 len 16384 data: 2 2 2 2 02/30 Write block 543 len 40960 data: 2 2 2 2 2 2 2 2 2 2 03/30 Write block 409 len 16384 data: 2 2 2 2 04/30 Write block 403 len 32768 data: 2 2 2 2 2 2 3 3 05/30 Write block 406 len 20480 data: 3 3 3 4 4 06/30 Read block 111 len 28672 07/30 Flush 08/30 Write block 319 len 12288 data: 2 2 2 09/30 Write block 404 len 4096 data: 3 10/30 Read block 209 len 36864 11/30 Write block 251 len 36864 data: 2 2 2 2 2 2 2 2 2 12/30 Read block 368 len 36864 13/30 Read block 132 len 20480 14/30 Write block 206 len 32768 data: 2 2 2 2 2 2 2 2 15/30 Read block 587 len 20480 16/30 Read block 332 len 24576 17/30 Read block 409 len 32768 18/30 Write block 483 len 32768 data: 2 2 2 2 2 2 2 2 19/30 Write block 239 len 32768 data: 2 2 2 2 2 2 2 2 20/30 Write block 576 len 4096 data: 2 21/30 Read block 577 len 36864 22/30 Read block 280 len 20480 23/30 Write block 198 len 40960 data: 2 2 2 2 2 2 2 2 3 3 24/30 Read block 536 len 36864 25/30 Read block 285 len 40960 26/30 Read block 117 len 16384 27/30 Write block 82 len 20480 data: 2 2 2 2 2 28/30 Read block 50 len 16384 29/30 Read block 462 len 16384 30/30 Write block 323 len 24576 data: 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:16 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 16 Acked 1015 Read 5 Done Done Done false 17 Acked 1016 Read 6 Done Done Done false 18 Acked 1017 Read 8 Done Done Done false 19 Acked 1018 Write 8 Done Done Done false 20 Acked 1019 Write 8 Done Done Done false 21 Acked 1020 Write 1 Done Done Done false 22 Acked 1021 Read 9 Done Done Done false 23 Acked 1022 Read 5 Done Done Done false 24 Acked 1023 Write 10 Done Done Done false 25 Acked 1024 Read 9 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Write 5 Done Done Done false 29 Acked 1028 Read 4 Done Done Done false 30 Acked 1029 Read 4 Done Done Done false 31 Acked 1030 Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 15 15 15 45 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1014 1014 1014 Downstairs last five completed: 1014 1013 1012 1011 1010 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:19.812600245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1193} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 1 1 1 1 1 1 T T T 5 100-119 1 1 1 1 1 1 F F F 6 120-139 1 1 1 1 1 1 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 T T T 10 200-219 2 2 2 4 4 4 T T T 11 220-239 1 1 1 1 1 1 T T T 12 240-259 2 2 2 4 4 4 T T T 13 260-279 1 1 1 2 2 2 F F F 14 280-299 1 1 1 2 2 2 F F F 15 300-319 2 2 2 4 4 4 F F F 16 320-339 2 2 2 4 4 4 T T T 17 340-359 1 1 1 2 2 2 F F F 18 360-379 1 1 1 2 2 2 F F F 19 380-399 1 1 1 2 2 2 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 1 1 1 2 2 2 F F F 22 440-459 2 2 2 3 3 3 F F F 23 460-479 2 2 2 3 3 3 F F F 24 480-499 1 1 1 2 2 2 T T T 25 500-519 1 1 1 2 2 2 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 2 2 2 3 3 3 F F F 28 560-579 1 1 1 1 1 1 T T T 29 580-599 1 1 1 1 1 1 F F F Max gen: 2, Max flush: 4 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 3 3 3 5 5 5 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 1 1 1 1 1 1 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 3 3 3 5 5 5 F F F 10 200-219 3 3 3 5 5 5 F F F 11 220-239 3 3 3 5 5 5 F F F 12 240-259 3 3 3 5 5 5 F F F 13 260-279 1 1 1 2 2 2 F F F 14 280-299 1 1 1 2 2 2 F F F 15 300-319 2 2 2 4 4 4 F F F 16 320-339 3 3 3 5 5 5 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 1 1 1 2 2 2 F F F 19 380-399 1 1 1 2 2 2 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 1 1 1 2 2 2 F F F 22 440-459 2 2 2 3 3 3 F F F 23 460-479 2 2 2 3 3 3 F F F 24 480-499 3 3 3 5 5 5 F F F 25 500-519 1 1 1 2 2 2 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 2 2 2 3 3 3 F F F 28 560-579 3 3 3 5 5 5 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 3, Max flush: 5 Begin loop 2 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.306711007Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307071121Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307090089Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201} {"msg":"Crucible e9b2cc0f-15bf-45b1-8f2b-6a5a17824305 has session id: 4f543617-6ed5-4c40-906a-c65b5920f72f","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307361362Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Upstairs opts: Upstairs UUID: e9b2cc0f-15bf-45b1-8f2b-6a5a17824305, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307396719Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Crucible stats registered with UUID: e9b2cc0f-15bf-45b1-8f2b-6a5a17824305","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307413497Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307712988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201} {"msg":"e9b2cc0f-15bf-45b1-8f2b-6a5a17824305 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307775661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307801969Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"0","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307818957Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"0","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307836335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"1","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307860463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"1","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307886461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307913348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307942955Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.307968832Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"io task","client":"0","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.30799399Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"io task","client":"1","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:35.308107348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:35.308157374Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"ds_connection connected from Ok(127.0.0.1:43215)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.308184201Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"io task","client":"1","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:35.308211588Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.308232826Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.308266463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"ds_connection connected from Ok(127.0.0.1:37207)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.30829924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"io task","client":"0","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.308482191Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.309068823Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"0","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.309095481Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"0","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.309111469Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"1","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.309129257Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"1","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.30970606Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"1","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.309841757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"0","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.310150376Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"1","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.310176234Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"1","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.310190742Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.310345987Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"0","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.310364915Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"0","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:35.310378394Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.309493826Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"ds_connection connected from Ok(127.0.0.1:44675)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.309643211Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"io task","client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.310729244Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.310758281Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.311336554Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.311919526Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.311947873Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 5, 1, 1, 1, 1, 5, 5, 5]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.311967511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 3, 1, 1, 1, 1, 3, 3, 3]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.311982499Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.311997118Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 5, 1, 1, 1, 1, 5, 5, 5]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312011316Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 3, 1, 1, 1, 1, 3, 3, 3]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312024865Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312039224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 5, 1, 1, 1, 1, 5, 5, 5]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312052943Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 3, 1, 1, 1, 1, 3, 3, 3]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312066451Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:23:45.3120806Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312097798Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312111047Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Next flush: 6","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312124085Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312136914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"":"downstairs","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312150273Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312165181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"0","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.31217887Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"1","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312192299Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"client":"2","":"downstairs","session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312205817Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312218236Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"e9b2cc0f-15bf-45b1-8f2b-6a5a17824305 is now active with session: 4f543617-6ed5-4c40-906a-c65b5920f72f","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312231205Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312243044Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201,"session_id":"4f543617-6ed5-4c40-906a-c65b5920f72f"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:45.312255282Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 433 len 12288 data: 2 2 2 02/30 Read block 246 len 40960 03/30 Write block 269 len 28672 data: 2 2 2 2 2 2 2 04/30 Write block 197 len 4096 data: 2 05/30 Read block 489 len 28672 06/30 Read block 409 len 20480 07/30 Write block 25 len 24576 data: 2 2 2 2 2 2 08/30 Write block 231 len 40960 data: 2 2 2 2 2 2 2 2 3 3 09/30 Write block 88 len 40960 data: 2 2 2 2 2 2 2 2 2 2 10/30 Read block 258 len 12288 11/30 Read block 142 len 8192 12/30 Write block 546 len 8192 data: 3 3 13/30 Write block 0 len 8192 data: 2 2 14/30 Read block 150 len 8192 15/30 Write block 226 len 4096 data: 2 16/30 Write block 477 len 20480 data: 2 2 2 2 2 17/30 Write block 31 len 20480 data: 2 2 2 2 2 18/30 Read block 332 len 12288 19/30 Read block 438 len 12288 20/30 Read block 197 len 32768 21/30 Write block 565 len 4096 data: 2 22/30 Read block 566 len 28672 23/30 Write block 190 len 24576 data: 2 2 2 2 2 2 24/30 Read block 107 len 16384 25/30 Read block 510 len 12288 26/30 Read block 335 len 20480 27/30 Read block 516 len 16384 28/30 Write block 179 len 8192 data: 2 2 29/30 Write block 368 len 12288 data: 2 2 2 30/30 Read block 360 len 28672 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Read 4 Done Done Done false 29 Acked 1028 Write 2 Done Done Sent false 30 Acked 1029 Write 3 Done Done Sent false 31 Acked 1030 Read 7 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 3 4 Done 5 6 3 14 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:45.39219013Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1201} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 6 6 6 F F F 1 020-039 4 4 4 6 6 6 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 4 4 4 6 6 6 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 1 1 1 1 1 1 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 T T T 9 180-199 4 4 4 6 6 6 T T T 10 200-219 3 3 3 5 5 5 F F F 11 220-239 4 4 4 6 6 6 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 4 4 4 6 6 6 F F F 14 280-299 1 1 1 2 2 2 F F F 15 300-319 2 2 2 4 4 4 F F F 16 320-339 3 3 3 5 5 5 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 1 1 1 2 2 2 T T T 19 380-399 1 1 1 2 2 2 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 4 4 4 6 6 6 F F F 22 440-459 2 2 2 3 3 3 F F F 23 460-479 4 4 4 6 6 6 F F F 24 480-499 4 4 4 6 6 6 F F F 25 500-519 1 1 1 2 2 2 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 4 4 4 6 6 6 F F F 28 560-579 4 4 4 6 6 6 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 4, Max flush: 6 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 6 6 6 F F F 1 020-039 4 4 4 6 6 6 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 4 4 4 6 6 6 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 1 1 1 1 1 1 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 5 5 5 7 7 7 F F F 9 180-199 5 5 5 7 7 7 F F F 10 200-219 3 3 3 5 5 5 F F F 11 220-239 4 4 4 6 6 6 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 4 4 4 6 6 6 F F F 14 280-299 1 1 1 2 2 2 F F F 15 300-319 2 2 2 4 4 4 F F F 16 320-339 3 3 3 5 5 5 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 5 5 5 7 7 7 F F F 19 380-399 1 1 1 2 2 2 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 4 4 4 6 6 6 F F F 22 440-459 2 2 2 3 3 3 F F F 23 460-479 4 4 4 6 6 6 F F F 24 480-499 4 4 4 6 6 6 F F F 25 500-519 1 1 1 2 2 2 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 4 4 4 6 6 6 F F F 28 560-579 4 4 4 6 6 6 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 5, Max flush: 7 Begin loop 3 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.866445699Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.866769137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.866787725Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209} {"msg":"Crucible e2a51f2c-a2a9-4116-8119-454039d3d67e has session id: d0dd3de4-f969-495e-8536-625c8b942b88","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867086685Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Upstairs opts: Upstairs UUID: e2a51f2c-a2a9-4116-8119-454039d3d67e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.86713263Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Crucible stats registered with UUID: e2a51f2c-a2a9-4116-8119-454039d3d67e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867155198Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.86753911Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209} {"msg":"e2a51f2c-a2a9-4116-8119-454039d3d67e active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867587485Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867615013Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.8676412Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867664068Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"1","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867685806Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"1","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867710523Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"2","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867737031Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"2","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867762538Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867793025Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"io task","client":"1","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867816583Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"io task","client":"2","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"ds_connection connected from Ok(127.0.0.1:40240)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.867957599Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"io task","client":"2","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:59.868000435Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"ds_connection connected from Ok(127.0.0.1:38029)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.868104054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"io task","client":"1","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:59.868132702Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:23:59.868153599Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.868166648Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.868180187Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.868193375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.868736411Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"2","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.868771138Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"2","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.868798745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"1","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.868826723Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"1","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.869410625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"1","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.869445971Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"2","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.869924204Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"2","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.869950461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"2","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.86996533Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.869989977Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"1","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.870011665Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"1","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:23:59.870032493Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.869321948Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"ds_connection connected from Ok(127.0.0.1:51302)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.869521069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"io task","client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.870595162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.870630279Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871159616Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871684434Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871712562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[0]R flush_numbers[0..12]: [6, 6, 1, 1, 6, 1, 1, 1, 7, 7, 5, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871732049Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[0]R generation[0..12]: [4, 4, 1, 1, 4, 1, 1, 1, 5, 5, 3, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871754847Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871773416Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[1]R flush_numbers[0..12]: [6, 6, 1, 1, 6, 1, 1, 1, 7, 7, 5, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871790754Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[1]R generation[0..12]: [4, 4, 1, 1, 4, 1, 1, 1, 5, 5, 3, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871807952Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.87182562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[2]R flush_numbers[0..12]: [6, 6, 1, 1, 6, 1, 1, 1, 7, 7, 5, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871842969Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} Wait for a query_work_queue command to finish before sending IO {"msg":"[2]R generation[0..12]: [4, 4, 1, 1, 4, 1, 1, 1, 5, 5, 3, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871863476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871881795Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Max found gen is 6","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871898463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Generation requested: 6 >= found:6","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871915231Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Next flush: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.87193172Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871947968Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"":"downstairs","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871964617Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871979775Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"0","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.871998743Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"1","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.872015481Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"client":"2","":"downstairs","session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.87203217Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.872047558Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"e2a51f2c-a2a9-4116-8119-454039d3d67e is now active with session: d0dd3de4-f969-495e-8536-625c8b942b88","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.872063577Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.872078595Z"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209,"session_id":"d0dd3de4-f969-495e-8536-625c8b942b88"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:09.872097803Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 207 len 12288 data: 4 3 3 02/30 Read block 399 len 36864 03/30 Read block 438 len 4096 04/30 Read block 588 len 8192 05/30 Write block 96 len 36864 data: 3 3 2 2 2 2 2 2 2 06/30 Write block 316 len 20480 data: 2 2 2 3 3 07/30 Write block 103 len 8192 data: 3 3 08/30 Read block 179 len 16384 09/30 Read block 456 len 24576 10/30 Read block 105 len 40960 11/30 Read block 90 len 28672 12/30 Write block 262 len 24576 data: 2 2 2 2 2 2 13/30 Write block 37 len 16384 data: 2 2 2 2 14/30 Read block 416 len 40960 15/30 Read block 312 len 24576 16/30 Write block 575 len 12288 data: 2 3 2 17/30 Write block 199 len 28672 data: 3 3 3 3 3 3 3 18/30 Read block 426 len 28672 19/30 Read block 162 len 24576 20/30 Write block 138 len 36864 data: 2 2 2 2 2 2 2 2 2 21/30 Flush 22/30 Write block 582 len 20480 data: 2 2 2 2 2 23/30 Write block 571 len 32768 data: 2 2 2 2 3 4 3 2 24/30 Write block 379 len 20480 data: 2 2 2 2 2 25/30 Read block 36 len 24576 26/30 Write block 567 len 8192 data: 2 2 27/30 Write block 550 len 36864 data: 3 3 3 2 2 2 2 2 2 28/30 Read block 423 len 24576 29/30 Read block 96 len 40960 30/30 Write block 284 len 24576 data: 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1 Acked 1000 Write 3 Done Done Done false 2 Acked 1001 Read 9 Done Done Done false 3 Acked 1002 Read 1 Done Done Done false 4 Acked 1003 Read 2 Done Done Done false 5 Acked 1004 Write 9 Done Done Done false 6 Acked 1005 Write 5 Done Done Done false 7 Acked 1006 Write 2 Done Done Done false 8 Acked 1007 Read 4 Done Done Done false 9 Acked 1008 Read 6 Done Done Done false 10 Acked 1009 Read 10 Done Done Done false 11 Acked 1010 Read 7 Done Done Done false 12 Acked 1011 Write 6 Done Done Done false 13 Acked 1012 Write 4 Done Done Done false 14 Acked 1013 Read 10 Done Done Done false 15 Acked 1014 Read 6 Done Done Done false 16 Acked 1015 Write 3 Done Done Done false 17 Acked 1016 Write 7 Done Done Done false 18 Acked 1017 Read 7 Done Done Done false 19 Acked 1018 Read 6 Done Done Done false 20 Acked 1019 Write 9 Done Done Done false 21 Acked 1020 Flush 0 Done Done Sent false 22 Acked 1021 Write 5 Done Done Sent false 23 Acked 1022 Write 8 Done Done Sent false 24 Acked 1023 Write 5 Done Done Sent false 25 Acked 1024 Read 6 Done Done Sent false 26 Acked 1025 Write 2 Done Done Sent false 27 Acked 1026 Write 9 Done Done Sent false 28 Acked 1027 Read 6 Done Done Sent false 29 Acked 1028 Read 10 Done Done Sent false 30 Acked 1029 Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 10 12 Done 29 29 20 78 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 0 Downstairs last five completed: Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:24:09.977989118Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1209} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 6 6 6 F F F 1 020-039 6 6 6 8 8 8 F F F 2 040-059 6 6 6 8 8 8 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 6 6 6 8 8 8 F F F 5 100-119 6 6 6 8 8 8 F F F 6 120-139 6 6 6 8 8 8 F F F 7 140-159 6 6 6 8 8 8 F F F 8 160-179 5 5 5 7 7 7 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 6 6 6 8 8 8 F F F 11 220-239 4 4 4 6 6 6 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 1 1 1 2 2 2 T T F <--- 15 300-319 6 6 6 8 8 8 F F F 16 320-339 6 6 6 8 8 8 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 5 5 5 7 7 7 T T F <--- 19 380-399 1 1 1 2 2 2 T T F <--- 20 400-419 2 2 2 4 4 4 F F F 21 420-439 4 4 4 6 6 6 F F F 22 440-459 2 2 2 3 3 3 F F F 23 460-479 4 4 4 6 6 6 F F F 24 480-499 4 4 4 6 6 6 F F F 25 500-519 1 1 1 2 2 2 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 4 4 4 6 6 6 T T F <--- 28 560-579 6 6 6 8 8 8 T T F <--- 29 580-599 1 1 1 1 1 1 T T F <--- Max gen: 6, Max flush: 8 Error: Difference in extent metadata found! A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 6 6 6 F F F 1 020-039 6 6 6 8 8 8 F F F 2 040-059 6 6 6 8 8 8 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 6 6 6 8 8 8 F F F 5 100-119 6 6 6 8 8 8 F F F 6 120-139 6 6 6 8 8 8 F F F 7 140-159 6 6 6 8 8 8 F F F 8 160-179 5 5 5 7 7 7 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 6 6 6 8 8 8 F F F 11 220-239 4 4 4 6 6 6 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 7 7 7 9 9 9 F F F 15 300-319 6 6 6 8 8 8 F F F 16 320-339 6 6 6 8 8 8 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 7 7 7 9 9 9 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 4 4 4 6 6 6 F F F 22 440-459 2 2 2 3 3 3 F F F 23 460-479 4 4 4 6 6 6 F F F 24 480-499 4 4 4 6 6 6 F F F 25 500-519 1 1 1 2 2 2 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 7 7 7 9 9 9 F F F 28 560-579 7 7 7 9 9 9 F F F 29 580-599 7 7 7 9 9 9 F F F Max gen: 7, Max flush: 9 Begin loop 4 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.972411094Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.972650481Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.972673948Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217} {"msg":"Crucible 8fc905a8-d550-4ab9-b674-bacef92eba05 has session id: c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973180048Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Upstairs opts: Upstairs UUID: 8fc905a8-d550-4ab9-b674-bacef92eba05, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973227463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Crucible stats registered with UUID: 8fc905a8-d550-4ab9-b674-bacef92eba05","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973255581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973540102Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217} {"msg":"8fc905a8-d550-4ab9-b674-bacef92eba05 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973635843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973656821Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973675629Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973704216Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"1","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973730913Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"1","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973754751Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"2","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973778529Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"2","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973802906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"io task","client":"2","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973827604Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"io task","client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973851082Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"io task","client":"1","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:47335)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973918725Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"io task","client":"1","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:33716)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.973950502Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"io task","client":"2","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:24:24.973980009Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"io task","client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:24:24.974007106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"io task","client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:24:24.974081749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.974110886Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.974134713Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.974310046Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"io task","client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.974743573Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"1","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.97477837Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"1","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.974806207Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"2","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.974833564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"2","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.975385859Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"1","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.975417276Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"2","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.975892689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"2","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.975917507Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"2","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.975932235Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.97598811Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"1","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.976013867Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"1","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:24.976033155Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.975466757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"io task","client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:55459)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.975890575Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"io task","client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.976934731Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.976977427Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.977526822Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978015674Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978043041Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[0]R flush_numbers[0..12]: [6, 8, 8, 1, 8, 8, 8, 8, 7, 8, 8, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978062169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[0]R generation[0..12]: [4, 6, 6, 1, 6, 6, 6, 6, 5, 6, 6, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978079967Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978098026Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[1]R flush_numbers[0..12]: [6, 8, 8, 1, 8, 8, 8, 8, 7, 8, 8, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978115144Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[1]R generation[0..12]: [4, 6, 6, 1, 6, 6, 6, 6, 5, 6, 6, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978132252Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978150011Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[2]R flush_numbers[0..12]: [6, 8, 8, 1, 8, 8, 8, 8, 7, 8, 8, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978167389Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} Wait for a query_work_queue command to finish before sending IO {"msg":"[2]R generation[0..12]: [4, 6, 6, 1, 6, 6, 6, 6, 5, 6, 6, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978187517Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978206025Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Max found gen is 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978222483Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978243971Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Next flush: 10","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978261009Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978277358Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"":"downstairs","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978294016Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978309195Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"0","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978325853Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"1","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978342381Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"client":"2","":"downstairs","session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.97835917Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978375428Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"8fc905a8-d550-4ab9-b674-bacef92eba05 is now active with session: c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978388367Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:34.978400566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217,"session_id":"c068bc70-bf0b-4fb3-b19e-1fbe8b1fb0ea"} {"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2024-07-09T21:24:34.978412894Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 488 len 40960 data: 3 3 3 2 2 2 2 2 2 2 02/30 Write block 23 len 28672 data: 2 2 3 3 3 3 3 03/30 Write block 300 len 4096 data: 2 04/30 Read block 7 len 40960 05/30 Flush 06/30 Write block 511 len 20480 data: 2 2 2 2 2 07/30 Flush 08/30 Write block 561 len 24576 data: 2 2 2 2 3 2 09/30 Read block 270 len 24576 10/30 Write block 132 len 28672 data: 2 2 2 2 2 2 3 11/30 Write block 547 len 20480 data: 4 3 3 4 4 12/30 Read block 287 len 20480 13/30 Read block 72 len 12288 14/30 Write block 78 len 28672 data: 2 2 2 2 3 3 3 15/30 Write block 293 len 28672 data: 2 2 2 2 2 2 2 16/30 Read block 408 len 32768 17/30 Read block 118 len 36864 18/30 Read block 318 len 28672 19/30 Flush 20/30 Write block 16 len 8192 data: 2 2 21/30 Write block 438 len 20480 data: 2 2 2 2 2 22/30 Read block 205 len 20480 23/30 Flush 24/30 Write block 28 len 40960 data: 4 4 3 3 3 3 3 3 2 3 25/30 Flush 26/30 Write block 458 len 24576 data: 2 3 3 3 3 2 27/30 Write block 493 len 24576 data: 3 3 3 3 3 2 28/30 Write block 206 len 28672 data: 4 5 4 4 3 3 3 29/30 Read block 56 len 12288 30/30 Read block 203 len 4096 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 6 Done Done Done false 28 Acked 1027 Write 6 Done Done Done false 29 Acked 1028 Write 7 Done Done Done false 30 Acked 1029 Read 3 Done Done Done false 31 Acked 1030 Read 1 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:24:35.060614618Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1217} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 8 8 8 14 14 14 F F F 1 020-039 8 8 8 15 15 15 F F F 2 040-059 6 6 6 8 8 8 F F F 3 060-079 8 8 8 12 12 12 F F F 4 080-099 8 8 8 12 12 12 F F F 5 100-119 6 6 6 8 8 8 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 6 6 6 8 8 8 F F F 8 160-179 5 5 5 7 7 7 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 6 6 6 8 8 8 T T T 11 220-239 4 4 4 6 6 6 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 8 8 8 12 12 12 F F F 15 300-319 8 8 8 10 10 10 F F F 16 320-339 6 6 6 8 8 8 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 7 7 7 9 9 9 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 8 8 8 14 14 14 F F F 22 440-459 8 8 8 14 14 14 T T T 23 460-479 4 4 4 6 6 6 T T T 24 480-499 8 8 8 10 10 10 T T T 25 500-519 8 8 8 11 11 11 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 8 8 8 12 12 12 F F F 28 560-579 8 8 8 12 12 12 F F F 29 580-599 7 7 7 9 9 9 F F F Max gen: 8, Max flush: 15 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 8 8 8 14 14 14 F F F 1 020-039 8 8 8 15 15 15 F F F 2 040-059 6 6 6 8 8 8 F F F 3 060-079 8 8 8 12 12 12 F F F 4 080-099 8 8 8 12 12 12 F F F 5 100-119 6 6 6 8 8 8 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 6 6 6 8 8 8 F F F 8 160-179 5 5 5 7 7 7 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 4 4 4 6 6 6 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 8 8 8 12 12 12 F F F 15 300-319 8 8 8 10 10 10 F F F 16 320-339 6 6 6 8 8 8 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 7 7 7 9 9 9 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 8 8 8 14 14 14 F F F 22 440-459 9 9 9 16 16 16 F F F 23 460-479 9 9 9 16 16 16 F F F 24 480-499 9 9 9 16 16 16 F F F 25 500-519 8 8 8 11 11 11 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 8 8 8 12 12 12 F F F 28 560-579 8 8 8 12 12 12 F F F 29 580-599 7 7 7 9 9 9 F F F Max gen: 9, Max flush: 16 Begin loop 5 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.591381077Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.591664579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.591690426Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225} {"msg":"Crucible 775b20f7-2ebd-4adb-8077-dbcdf8d25557 has session id: 2c07de82-153a-418e-8a1b-bf746191fbdf","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592172669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Upstairs opts: Upstairs UUID: 775b20f7-2ebd-4adb-8077-dbcdf8d25557, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592219614Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Crucible stats registered with UUID: 775b20f7-2ebd-4adb-8077-dbcdf8d25557","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592242512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592626394Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225} {"msg":"775b20f7-2ebd-4adb-8077-dbcdf8d25557 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592656511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592680588Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592704306Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592726014Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"1","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592754301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"1","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592778349Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"2","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592804416Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"2","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592831133Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"io task","client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.59285909Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"io task","client":"1","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.592884898Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"io task","client":"2","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:24:49.592946182Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"io task","client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:24:49.592978859Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"io task","client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"ds_connection connected from Ok(127.0.0.1:40806)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.593023744Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"io task","client":"2","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"ds_connection connected from Ok(127.0.0.1:37910)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.593051671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"io task","client":"1","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:24:49.593158311Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.593184528Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.593199657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.593349982Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"io task","client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.593891478Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"1","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.593922485Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"1","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.593948693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"2","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.59397611Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"2","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.59457855Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"1","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.594617976Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"2","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.595166822Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"2","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.595192779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"2","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.595208328Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.595221536Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"1","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.595235305Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"1","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:49.595248664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.593902091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"io task","client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"ds_connection connected from Ok(127.0.0.1:40609)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.594172125Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"io task","client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.595186944Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.595220901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.59573378Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596255908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596284305Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[0]R flush_numbers[0..12]: [14, 15, 8, 12, 12, 8, 12, 8, 7, 8, 16, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596302634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[0]R generation[0..12]: [8, 8, 6, 8, 8, 6, 8, 6, 5, 6, 9, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596320292Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.59633812Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[1]R flush_numbers[0..12]: [14, 15, 8, 12, 12, 8, 12, 8, 7, 8, 16, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596355368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[1]R generation[0..12]: [8, 8, 6, 8, 8, 6, 8, 6, 5, 6, 9, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596372167Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596389625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[2]R flush_numbers[0..12]: [14, 15, 8, 12, 12, 8, 12, 8, 7, 8, 16, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596406593Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[2]R generation[0..12]: [8, 8, 6, 8, 8, 6, 8, 6, 5, 6, 9, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596423222Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596445059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596461188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596477766Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Next flush: 17","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596498994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596515822Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"":"downstairs","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596532331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596547229Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"0","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596563638Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"1","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596580196Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"client":"2","":"downstairs","session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596596744Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596611883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"775b20f7-2ebd-4adb-8077-dbcdf8d25557 is now active with session: 2c07de82-153a-418e-8a1b-bf746191fbdf","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596628111Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596655948Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225,"session_id":"2c07de82-153a-418e-8a1b-bf746191fbdf"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:24:59.596672137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 94 len 40960 data: 3 3 4 4 3 3 3 3 3 4 02/30 Read block 297 len 8192 03/30 Write block 13 len 36864 data: 2 2 2 3 3 2 2 2 2 04/30 Read block 445 len 40960 05/30 Read block 454 len 4096 06/30 Read block 331 len 32768 07/30 Read block 225 len 20480 08/30 Read block 540 len 40960 09/30 Read block 429 len 8192 10/30 Read block 323 len 24576 11/30 Flush 12/30 Write block 440 len 36864 data: 3 3 3 2 2 2 2 2 2 13/30 Read block 421 len 28672 14/30 Read block 538 len 12288 15/30 Read block 373 len 40960 16/30 Read block 293 len 12288 17/30 Read block 292 len 8192 18/30 Read block 332 len 4096 19/30 Read block 510 len 36864 20/30 Read block 318 len 24576 21/30 Read block 559 len 32768 22/30 Write block 438 len 40960 data: 3 3 4 4 4 3 3 3 3 3 23/30 Read block 506 len 32768 24/30 Read block 166 len 20480 25/30 Read block 508 len 20480 26/30 Read block 175 len 24576 27/30 Write block 561 len 24576 data: 3 3 3 3 4 3 28/30 Write block 594 len 24576 data: 2 2 2 2 2 2 29/30 Read block 77 len 32768 30/30 Write block 523 len 28672 data: 2 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Read 8 Done Done Done false 25 Acked 1024 Read 5 Done Done Done false 26 Acked 1025 Read 5 Done Done Done false 27 Acked 1026 Read 6 Done Done Done false 28 Acked 1027 Write 6 Done Done Done false 29 Acked 1028 Write 6 Done Done Done false 30 Acked 1029 Read 8 Done Done Done false 31 Acked 1030 Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:24:59.622040452Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1225} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 17 17 17 F F F 1 020-039 10 10 10 17 17 17 F F F 2 040-059 6 6 6 8 8 8 F F F 3 060-079 8 8 8 12 12 12 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 6 6 6 8 8 8 F F F 8 160-179 5 5 5 7 7 7 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 4 4 4 6 6 6 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 8 8 8 12 12 12 F F F 15 300-319 8 8 8 10 10 10 F F F 16 320-339 6 6 6 8 8 8 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 7 7 7 9 9 9 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 8 8 8 14 14 14 T T T 22 440-459 10 10 10 18 18 18 T T T 23 460-479 9 9 9 16 16 16 F F F 24 480-499 9 9 9 16 16 16 F F F 25 500-519 8 8 8 11 11 11 F F F 26 520-539 1 1 1 1 1 1 T T T 27 540-559 8 8 8 12 12 12 F F F 28 560-579 8 8 8 12 12 12 T T T 29 580-599 7 7 7 9 9 9 T T T Max gen: 10, Max flush: 18 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 17 17 17 F F F 1 020-039 10 10 10 17 17 17 F F F 2 040-059 6 6 6 8 8 8 F F F 3 060-079 8 8 8 12 12 12 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 6 6 6 8 8 8 F F F 8 160-179 5 5 5 7 7 7 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 9 9 9 16 16 16 F F F 11 220-239 4 4 4 6 6 6 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 8 8 8 12 12 12 F F F 15 300-319 8 8 8 10 10 10 F F F 16 320-339 6 6 6 8 8 8 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 7 7 7 9 9 9 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 11 11 11 19 19 19 F F F 22 440-459 11 11 11 19 19 19 F F F 23 460-479 9 9 9 16 16 16 F F F 24 480-499 9 9 9 16 16 16 F F F 25 500-519 8 8 8 11 11 11 F F F 26 520-539 11 11 11 19 19 19 F F F 27 540-559 8 8 8 12 12 12 F F F 28 560-579 11 11 11 19 19 19 F F F 29 580-599 11 11 11 19 19 19 F F F Max gen: 11, Max flush: 19 Begin loop 6 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.239163613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.239446335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.239472353Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233} {"msg":"Crucible 5bb1d9e6-8e80-40ab-a71a-f63387ee8fc5 has session id: 570ee41b-d68b-401f-a4f3-762fc05b5d76","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.239690161Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Upstairs opts: Upstairs UUID: 5bb1d9e6-8e80-40ab-a71a-f63387ee8fc5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.239734057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Crucible stats registered with UUID: 5bb1d9e6-8e80-40ab-a71a-f63387ee8fc5","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.239760004Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240174253Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233} {"msg":"5bb1d9e6-8e80-40ab-a71a-f63387ee8fc5 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.24020589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240229988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"0","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240253675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"0","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240275763Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.24030635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240333547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"2","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240357835Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"2","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240380563Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"io task","client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240397901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"io task","client":"2","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240416989Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"io task","client":"0","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:25:14.240594101Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"io task","client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"ds_connection connected from Ok(127.0.0.1:57305)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240631318Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"io task","client":"2","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:25:14.240661765Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"io task","client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:25:14.240692712Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240718389Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240791352Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"ds_connection connected from Ok(127.0.0.1:52279)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240820539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"io task","client":"0","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.240964465Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"io task","client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.241561656Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"2","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.241591693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"2","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.241607541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"0","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.24162109Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"0","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.242195233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"0","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.242305352Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"2","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.242675635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"0","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.242702033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"0","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.242716731Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.242741729Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"2","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.242756547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"2","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:14.242770166Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.241482718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"io task","client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"ds_connection connected from Ok(127.0.0.1:62731)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.241704256Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"io task","client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.242736744Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.2427763Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.24327621Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243799828Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243820216Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[0]R flush_numbers[0..12]: [17, 17, 8, 12, 17, 17, 12, 8, 7, 8, 16, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243838625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[0]R generation[0..12]: [10, 10, 6, 8, 10, 10, 8, 6, 5, 6, 9, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243852843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243867571Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[1]R flush_numbers[0..12]: [17, 17, 8, 12, 17, 17, 12, 8, 7, 8, 16, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.24388128Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[1]R generation[0..12]: [10, 10, 6, 8, 10, 10, 8, 6, 5, 6, 9, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243896269Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243910537Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[2]R flush_numbers[0..12]: [17, 17, 8, 12, 17, 17, 12, 8, 7, 8, 16, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243924246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[2]R generation[0..12]: [10, 10, 6, 8, 10, 10, 8, 6, 5, 6, 9, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243937805Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243951853Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.243964902Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Generation requested: 12 >= found:12","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2024-07-09T21:25:24.24398145Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Next flush: 20","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244000158Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244025656Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"":"downstairs","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244044094Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244059232Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"0","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244075951Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"1","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244092419Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"client":"2","":"downstairs","session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244109208Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244124586Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"5bb1d9e6-8e80-40ab-a71a-f63387ee8fc5 is now active with session: 570ee41b-d68b-401f-a4f3-762fc05b5d76","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244140715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244155503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233,"session_id":"570ee41b-d68b-401f-a4f3-762fc05b5d76"} {"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:24.244170632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 54 len 16384 data: 2 2 2 2 03/30 Flush 04/30 Write block 388 len 32768 data: 2 2 2 2 2 2 2 2 05/30 Read block 545 len 36864 06/30 Read block 413 len 16384 07/30 Read block 256 len 16384 08/30 Read block 151 len 36864 09/30 Read block 53 len 12288 10/30 Read block 258 len 28672 11/30 Flush 12/30 Write block 580 len 32768 data: 2 2 3 3 3 3 3 2 13/30 Flush 14/30 Write block 325 len 40960 data: 3 3 3 3 2 2 2 2 2 2 15/30 Write block 383 len 28672 data: 3 2 2 2 2 3 3 16/30 Write block 146 len 40960 data: 3 2 2 2 2 2 2 2 2 2 17/30 Read block 471 len 36864 18/30 Read block 73 len 28672 19/30 Write block 388 len 36864 data: 4 4 3 3 3 3 3 3 2 20/30 Write block 206 len 24576 data: 5 6 5 5 4 4 21/30 Read block 261 len 40960 22/30 Read block 468 len 20480 23/30 Read block 94 len 4096 24/30 Read block 333 len 16384 25/30 Write block 306 len 24576 data: 2 2 2 2 2 2 26/30 Read block 78 len 20480 27/30 Read block 375 len 4096 28/30 Read block 42 len 40960 29/30 Write block 312 len 12288 data: 2 2 2 30/30 Write block 59 len 4096 data: 2 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 9 Done Done Done false 21 Acked 1020 Write 6 Done Done Done false 22 Acked 1021 Read 10 Done Done Done false 23 Acked 1022 Read 5 Done Done Done false 24 Acked 1023 Read 1 Done Done Done false 25 Acked 1024 Read 4 Done Done Done false 26 Acked 1025 Write 6 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Write 3 Sent Sent Sent false 31 Acked 1030 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 2 2 6 Done 10 10 10 30 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:25:24.297926474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1233} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 17 17 17 F F F 1 020-039 10 10 10 17 17 17 F F F 2 040-059 12 12 12 21 21 21 T T T 3 060-079 8 8 8 12 12 12 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 12 12 12 24 24 24 F F F 8 160-179 5 5 5 7 7 7 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 9 9 9 16 16 16 T T T 11 220-239 4 4 4 6 6 6 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 8 8 8 12 12 12 F F F 15 300-319 8 8 8 10 10 10 T T T 16 320-339 12 12 12 24 24 24 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 12 12 12 24 24 24 T T T 20 400-419 2 2 2 4 4 4 F F F 21 420-439 11 11 11 19 19 19 F F F 22 440-459 11 11 11 19 19 19 F F F 23 460-479 9 9 9 16 16 16 F F F 24 480-499 9 9 9 16 16 16 F F F 25 500-519 8 8 8 11 11 11 F F F 26 520-539 11 11 11 19 19 19 F F F 27 540-559 8 8 8 12 12 12 F F F 28 560-579 11 11 11 19 19 19 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 12, Max flush: 24 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 17 17 17 F F F 1 020-039 10 10 10 17 17 17 F F F 2 040-059 13 13 13 25 25 25 F F F 3 060-079 8 8 8 12 12 12 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 12 12 12 24 24 24 F F F 8 160-179 5 5 5 7 7 7 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 13 13 13 25 25 25 F F F 11 220-239 4 4 4 6 6 6 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 8 8 8 12 12 12 F F F 15 300-319 13 13 13 25 25 25 F F F 16 320-339 12 12 12 24 24 24 F F F 17 340-359 1 1 1 2 2 2 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 13 13 13 25 25 25 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 11 11 11 19 19 19 F F F 22 440-459 11 11 11 19 19 19 F F F 23 460-479 9 9 9 16 16 16 F F F 24 480-499 9 9 9 16 16 16 F F F 25 500-519 8 8 8 11 11 11 F F F 26 520-539 11 11 11 19 19 19 F F F 27 540-559 8 8 8 12 12 12 F F F 28 560-579 11 11 11 19 19 19 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 13, Max flush: 25 Begin loop 7 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.81721048Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.817523129Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.817554296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241} {"msg":"Crucible 62b7bc97-17dd-4339-a5d5-e8baed5f3634 has session id: a548f762-b4ea-4cc5-89fe-84cab0772cb0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.817939818Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Upstairs opts: Upstairs UUID: 62b7bc97-17dd-4339-a5d5-e8baed5f3634, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.817981264Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Crucible stats registered with UUID: 62b7bc97-17dd-4339-a5d5-e8baed5f3634","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818004901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818336448Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241} {"msg":"62b7bc97-17dd-4339-a5d5-e8baed5f3634 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818399782Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.81842407Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818439508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818453787Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"1","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818472355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"1","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818497632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"2","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.81852334Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"2","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818547368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"io task","client":"2","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818591693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"io task","client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818617921Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"io task","client":"1","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"ds_connection connected from Ok(127.0.0.1:42538)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818812491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"io task","client":"2","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"ds_connection connected from Ok(127.0.0.1:46120)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.818893843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"io task","client":"1","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:25:38.819016111Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"io task","client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:25:38.819041479Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"io task","client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:25:38.819183844Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.819213802Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.819236279Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.819258607Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"io task","client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.819692864Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"2","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.819747169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"2","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.819778256Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"1","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.819806403Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"1","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.820394955Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"2","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.820420812Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"1","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.820954269Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"2","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.820977827Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"2","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.820992605Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.821005224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"1","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.821019403Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"1","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:38.821032881Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.821187441Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"io task","client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"ds_connection connected from Ok(127.0.0.1:52513)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.821511659Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"io task","client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.822466694Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.822491032Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823073034Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823642838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823666495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[0]R flush_numbers[0..12]: [17, 17, 25, 12, 17, 17, 12, 24, 7, 8, 25, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823681464Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[0]R generation[0..12]: [10, 10, 13, 8, 10, 10, 8, 12, 5, 6, 13, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823695552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823710061Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[1]R flush_numbers[0..12]: [17, 17, 25, 12, 17, 17, 12, 24, 7, 8, 25, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.82372413Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[1]R generation[0..12]: [10, 10, 13, 8, 10, 10, 8, 12, 5, 6, 13, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823738588Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823753007Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[2]R flush_numbers[0..12]: [17, 17, 25, 12, 17, 17, 12, 24, 7, 8, 25, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823767645Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[2]R generation[0..12]: [10, 10, 13, 8, 10, 10, 8, 12, 5, 6, 13, 4]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823785523Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823803082Z"Wait for a query_work_queue command to finish before sending IO ,"hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Max found gen is 14","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.82382387Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823840678Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Next flush: 26","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823860576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823877405Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"":"downstairs","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823894063Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823909041Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"0","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.82392565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"1","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823942128Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"client":"2","":"downstairs","session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823959376Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823974755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"62b7bc97-17dd-4339-a5d5-e8baed5f3634 is now active with session: a548f762-b4ea-4cc5-89fe-84cab0772cb0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.823990913Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.824005952Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241,"session_id":"a548f762-b4ea-4cc5-89fe-84cab0772cb0"} {"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:25:48.82402147Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 353 len 24576 data: 2 2 2 2 2 2 02/30 Write block 171 len 28672 data: 2 2 2 2 2 2 2 03/30 Read block 389 len 4096 04/30 Read block 170 len 32768 05/30 Read block 24 len 40960 06/30 Read block 268 len 12288 07/30 Read block 554 len 20480 08/30 Flush 09/30 Flush 10/30 Write block 508 len 40960 data: 2 2 2 3 3 3 3 3 2 2 11/30 Write block 67 len 32768 data: 2 2 2 2 2 2 2 2 12/30 Read block 225 len 40960 13/30 Read block 453 len 40960 14/30 Flush 15/30 Write block 385 len 40960 data: 3 3 3 5 5 4 4 4 4 4 16/30 Write block 571 len 36864 data: 3 3 3 3 4 5 4 3 2 17/30 Write block 317 len 24576 data: 3 3 4 4 3 2 18/30 Flush 19/30 Write block 466 len 36864 data: 2 2 2 2 2 2 2 2 2 20/30 Read block 339 len 32768 21/30 Read block 211 len 16384 22/30 Flush 23/30 Write block 234 len 4096 data: 3 24/30 Read block 106 len 40960 25/30 Write block 296 len 12288 data: 3 3 3 26/30 Read block 321 len 16384 27/30 Read block 274 len 20480 28/30 Read block 28 len 24576 29/30 Read block 18 len 24576 30/30 Write block 510 len 8192 data: 3 4 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 1 Done Done Done false 25 Acked 1024 Read 10 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Read 5 Done Done Done false 29 Acked 1028 Read 6 Done Done Done false 30 Acked 1029 Read 6 Done Done Done false 31 Acked 1030 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:25:48.882162728Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1241} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 17 17 17 F F F 1 020-039 10 10 10 17 17 17 F F F 2 040-059 13 13 13 25 25 25 F F F 3 060-079 14 14 14 28 28 28 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 12 12 12 24 24 24 F F F 8 160-179 14 14 14 26 26 26 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 13 13 13 25 25 25 F F F 11 220-239 4 4 4 6 6 6 T T T 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 8 8 8 12 12 12 T T T 15 300-319 14 14 14 30 30 30 F F F 16 320-339 14 14 14 30 30 30 F F F 17 340-359 14 14 14 26 26 26 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 14 14 14 30 30 30 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 11 11 11 19 19 19 F F F 22 440-459 11 11 11 19 19 19 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 9 9 9 16 16 16 F F F 25 500-519 14 14 14 28 28 28 T T T 26 520-539 11 11 11 19 19 19 F F F 27 540-559 8 8 8 12 12 12 F F F 28 560-579 14 14 14 30 30 30 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 14, Max flush: 31 A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 17 17 17 F F F 1 020-039 10 10 10 17 17 17 F F F 2 040-059 13 13 13 25 25 25 F F F 3 060-079 14 14 14 28 28 28 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 12 12 12 24 24 24 F F F 8 160-179 14 14 14 26 26 26 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 13 13 13 25 25 25 F F F 11 220-239 15 15 15 32 32 32 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 15 15 15 32 32 32 F F F 15 300-319 14 14 14 30 30 30 F F F 16 320-339 14 14 14 30 30 30 F F F 17 340-359 14 14 14 26 26 26 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 14 14 14 30 30 30 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 11 11 11 19 19 19 F F F 22 440-459 11 11 11 19 19 19 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 9 9 9 16 16 16 F F F 25 500-519 15 15 15 32 32 32 F F F 26 520-539 11 11 11 19 19 19 F F F 27 540-559 8 8 8 12 12 12 F F F 28 560-579 14 14 14 30 30 30 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 15, Max flush: 32 Begin loop 8 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.286396477Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.286665701Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.286683539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249} {"msg":"Crucible 04a3fbd4-f3e3-4b4b-8117-66a19a524188 has session id: a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.286964261Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Upstairs opts: Upstairs UUID: 04a3fbd4-f3e3-4b4b-8117-66a19a524188, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287011497Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Crucible stats registered with UUID: 04a3fbd4-f3e3-4b4b-8117-66a19a524188","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287035504Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287366071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249} {"msg":"04a3fbd4-f3e3-4b4b-8117-66a19a524188 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287447923Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287473081Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"0","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287489419Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"0","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287503508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287517246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287534814Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"2","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287548923Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"2","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287562492Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"io task","client":"2","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287576461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"io task","client":"0","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287590349Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"io task","client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:03.287858322Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"io task","client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"ds_connection connected from Ok(127.0.0.1:48053)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.287905988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"io task","client":"0","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:03.287978091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"io task","client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:03.288012557Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.288033955Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.288057953Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"ds_connection connected from Ok(127.0.0.1:49785)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.28808192Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"io task","client":"2","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.288175731Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"io task","client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.288781181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"0","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.288816638Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"0","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.288844305Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"2","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.288870912Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"2","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.289391401Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"2","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.289531597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"0","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.289911379Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"2","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.289933847Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"2","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.289947925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.2900033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"0","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.290021778Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"0","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:03.290035267Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.289297646Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"io task","client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"ds_connection connected from Ok(127.0.0.1:55697)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.289480128Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"io task","client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.290589508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.290624554Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291127384Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291667421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291692858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[0]R flush_numbers[0..12]: [17, 17, 25, 28, 17, 17, 12, 24, 26, 8, 25, 32]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291710916Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[0]R generation[0..12]: [10, 10, 13, 14, 10, 10, 8, 12, 14, 6, 13, 15]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291728345Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291746243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[1]R flush_numbers[0..12]: [17, 17, 25, 28, 17, 17, 12, 24, 26, 8, 25, 32]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291763541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[1]R generation[0..12]: [10, 10, 13, 14, 10, 10, 8, 12, 14, 6, 13, 15]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.29178074Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291798678Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[2]R flush_numbers[0..12]: [17, 17, 25, 28, 17, 17, 12, 24, 26, 8, 25, 32]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291816006Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[2]R generation[0..12]: [10, 10, 13, 14, 10, 10, 8, 12, 14, 6, 13, 15]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291833244Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291850843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291867231Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291883979Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Next flush: 33","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291900418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291919766Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G"Wait for a query_work_queue command to finish before sending IO ,"pid":1249,"":"downstairs","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291940494Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.291955472Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"0","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.29198279Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"1","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.292014286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"client":"2","":"downstairs","session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.292046553Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.292072791Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"04a3fbd4-f3e3-4b4b-8117-66a19a524188 is now active with session: a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.292102508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.292129815Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249,"session_id":"a3498b32-c0ed-4a9a-a9e1-0852c6aeac9a"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:13.292156532Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 538 len 20480 data: 2 2 2 2 2 02/30 Write block 52 len 28672 data: 2 2 3 3 3 3 2 03/30 Write block 523 len 24576 data: 3 3 3 3 3 3 04/30 Flush 05/30 Write block 11 len 4096 data: 2 06/30 Write block 322 len 4096 data: 3 07/30 Write block 321 len 40960 data: 4 4 3 3 4 4 4 4 3 3 08/30 Read block 223 len 36864 09/30 Flush 10/30 Write block 449 len 40960 data: 2 2 2 2 2 2 2 2 2 3 11/30 Read block 538 len 4096 12/30 Write block 462 len 36864 data: 4 3 2 2 3 3 3 3 3 13/30 Write block 427 len 28672 data: 2 2 2 2 2 2 3 14/30 Read block 411 len 8192 15/30 Write block 486 len 24576 data: 3 3 4 4 4 3 16/30 Read block 386 len 32768 17/30 Flush 18/30 Write block 297 len 20480 data: 4 4 3 3 2 19/30 Read block 104 len 36864 20/30 Write block 155 len 16384 data: 3 2 2 2 21/30 Write block 36 len 16384 data: 3 4 3 3 22/30 Read block 138 len 8192 23/30 Read block 394 len 28672 24/30 Write block 43 len 4096 data: 2 25/30 Read block 533 len 24576 26/30 Read block 64 len 32768 27/30 Write block 288 len 8192 data: 3 3 28/30 Read block 169 len 32768 29/30 Read block 309 len 28672 30/30 Read block 75 len 12288 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 5 Done Done Done false 19 Acked 1018 Read 9 Done Done Done false 20 Acked 1019 Write 4 Done Done Done false 21 Acked 1020 Write 4 Done Done Done false 22 Acked 1021 Read 2 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Write 1 Done Done Done false 25 Acked 1024 Read 6 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Read 8 Done Done Done false 29 Acked 1028 Read 7 Done Done Done false 30 Acked 1029 Read 3 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 0 2 Done 12 12 13 37 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:13.338233256Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1249} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 34 34 34 F F F 1 020-039 10 10 10 17 17 17 T T T 2 040-059 16 16 16 33 33 33 T T T 3 060-079 14 14 14 28 28 28 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 12 12 12 24 24 24 T T T 8 160-179 14 14 14 26 26 26 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 13 13 13 25 25 25 F F F 11 220-239 15 15 15 32 32 32 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 15 15 15 32 32 32 T T T 15 300-319 14 14 14 30 30 30 T T T 16 320-339 16 16 16 34 34 34 F F F 17 340-359 14 14 14 26 26 26 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 14 14 14 30 30 30 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 16 16 16 35 35 35 F F F 22 440-459 16 16 16 35 35 35 F F F 23 460-479 16 16 16 35 35 35 F F F 24 480-499 16 16 16 35 35 35 F F F 25 500-519 15 15 15 32 32 32 F F F 26 520-539 16 16 16 33 33 33 F F F 27 540-559 16 16 16 33 33 33 F F F 28 560-579 14 14 14 30 30 30 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 16, Max flush: 35 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 34 34 34 F F F 1 020-039 17 17 17 36 36 36 F F F 2 040-059 17 17 17 36 36 36 F F F 3 060-079 14 14 14 28 28 28 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 14 14 14 26 26 26 F F F 9 180-199 6 6 6 8 8 8 F F F 10 200-219 13 13 13 25 25 25 F F F 11 220-239 15 15 15 32 32 32 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 17 17 17 36 36 36 F F F 15 300-319 17 17 17 36 36 36 F F F 16 320-339 16 16 16 34 34 34 F F F 17 340-359 14 14 14 26 26 26 F F F 18 360-379 7 7 7 9 9 9 F F F 19 380-399 14 14 14 30 30 30 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 16 16 16 35 35 35 F F F 22 440-459 16 16 16 35 35 35 F F F 23 460-479 16 16 16 35 35 35 F F F 24 480-499 16 16 16 35 35 35 F F F 25 500-519 15 15 15 32 32 32 F F F 26 520-539 16 16 16 33 33 33 F F F 27 540-559 16 16 16 33 33 33 F F F 28 560-579 14 14 14 30 30 30 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 17, Max flush: 36 Begin loop 9 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.952478511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.952828156Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.952852244Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257} {"msg":"Crucible 61ac8675-3bc3-47f1-a850-e4b4e6538eba has session id: eb5436d2-1d46-411a-8503-499d557f1913","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953185111Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Upstairs opts: Upstairs UUID: 61ac8675-3bc3-47f1-a850-e4b4e6538eba, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953218497Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Crucible stats registered with UUID: 61ac8675-3bc3-47f1-a850-e4b4e6538eba","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953234266Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953573012Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257} {"msg":"61ac8675-3bc3-47f1-a850-e4b4e6538eba active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953608359Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953623667Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953638306Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953652174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"1","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953669353Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"1","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953683071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"2","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.95369665Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"2","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953756354Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"io task","client":"2","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953775272Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"io task","client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.953789531Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"io task","client":"1","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:27.953922748Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"io task","client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:27.953982072Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"io task","client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:27.954011149Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.954033537Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.954056474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"ds_connection connected from Ok(127.0.0.1:64801)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.954080132Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"io task","client":"1","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"ds_connection connected from Ok(127.0.0.1:57146)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.954105799Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"io task","client":"2","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.95430582Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"io task","client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.954748266Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"1","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.954775683Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"1","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.954791762Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"2","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.954852245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"2","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.955347486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"2","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.955472554Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"1","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.955872634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"2","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.955906251Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"2","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.955929429Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.955950057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"1","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.955972174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"1","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:27.955994272Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.955191796Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"io task","client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"ds_connection connected from Ok(127.0.0.1:57948)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.955378547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"io task","client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.95645902Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.956495607Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957143963Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957711976Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957737224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[0]R flush_numbers[0..12]: [34, 36, 36, 28, 17, 17, 12, 36, 26, 8, 25, 32]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957755782Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[0]R generation[0..12]: [16, 17, 17, 14, 10, 10, 8, 17, 14, 6, 13, 15]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.95777344Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957791828Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[1]R flush_numbers[0..12]: [34, 36, 36, 28, 17, 17, 12, 36, 26, 8, 25, 32]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957809426Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[1]R generation[0..12]: [16, 17, 17, 14, 10, 10, 8, 17, 14, 6, 13, 15]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957826795Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957844633Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[2]R flush_numbers[0..12]: [34, 36, 36, 28, 17, 17, 12, 36, 26, 8, 25, 32]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957862021Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[2]R generation[0..12]: [16, 17, 17, 14, 10, 10, 8, 17, 14, 6, 13, 15]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:26:37.957879159Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957900128Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Max found gen is 18","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957916656Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Generation requested: 18 >= found:18","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957933514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Next flush: 37","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957953452Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.95797033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"":"downstairs","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.957987069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.958002187Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"0","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.958019006Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"1","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.958035484Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"client":"2","":"downstairs","session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.958052492Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.958067761Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"61ac8675-3bc3-47f1-a850-e4b4e6538eba is now active with session: eb5436d2-1d46-411a-8503-499d557f1913","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.958083869Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.958098948Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257,"session_id":"eb5436d2-1d46-411a-8503-499d557f1913"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:37.958114476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 324 len 36864 data: 4 5 5 5 5 4 4 3 3 02/30 Write block 315 len 28672 data: 2 3 4 4 5 5 5 03/30 Read block 80 len 36864 04/30 Read block 61 len 36864 05/30 Write block 126 len 8192 data: 2 2 06/30 Write block 391 len 20480 data: 5 5 5 5 4 07/30 Flush 08/30 Write block 132 len 20480 data: 3 3 3 3 3 09/30 Read block 60 len 12288 10/30 Write block 188 len 28672 data: 2 2 3 3 3 3 3 11/30 Read block 522 len 36864 12/30 Write block 353 len 40960 data: 3 3 3 3 3 3 2 2 2 2 13/30 Write block 295 len 8192 data: 3 4 14/30 Read block 445 len 36864 15/30 Write block 373 len 12288 data: 2 2 2 16/30 Write block 14 len 20480 data: 3 3 4 4 3 17/30 Write block 205 len 16384 data: 4 6 7 6 18/30 Write block 443 len 36864 data: 4 4 4 4 4 3 3 3 3 19/30 Write block 556 len 28672 data: 3 3 3 2 2 4 4 20/30 Read block 322 len 36864 21/30 Read block 73 len 8192 22/30 Read block 370 len 40960 23/30 Read block 12 len 12288 24/30 Write block 434 len 20480 data: 3 3 2 2 4 25/30 Write block 556 len 36864 data: 4 4 4 3 3 5 5 4 4 26/30 Read block 570 len 40960 27/30 Read block 356 len 8192 28/30 Read block 66 len 24576 29/30 Write block 219 len 24576 data: 2 2 2 2 2 2 30/30 Read block 354 len 20480 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:16 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 16 Acked 1015 Write 3 Done Done Done false 17 Acked 1016 Write 5 Done Done Done false 18 Acked 1017 Write 4 Done Done Done false 19 Acked 1018 Write 9 Done Done Done false 20 Acked 1019 Write 7 Done Done Done false 21 Acked 1020 Read 9 Done Done Done false 22 Acked 1021 Read 2 Done Done Done false 23 Acked 1022 Read 10 Done Done Done false 24 Acked 1023 Read 3 Done Done Done false 25 Acked 1024 Write 5 Done Done Done false 26 Acked 1025 Write 9 Done Done Done false 27 Acked 1026 Read 10 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Read 6 Done Done Done false 30 Acked 1029 Write 6 Done Done Done false 31 Acked 1030 Read 5 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 16 16 16 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1014 1014 1014 Downstairs last five completed: 1014 1013 1012 1011 1010 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:38.024213355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1257} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 34 34 34 T T T 1 020-039 17 17 17 36 36 36 F F F 2 040-059 17 17 17 36 36 36 F F F 3 060-079 14 14 14 28 28 28 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 18 18 18 38 38 38 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 14 14 14 26 26 26 F F F 9 180-199 18 18 18 38 38 38 F F F 10 200-219 13 13 13 25 25 25 T T T 11 220-239 15 15 15 32 32 32 T T T 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 18 18 18 38 38 38 F F F 15 300-319 18 18 18 37 37 37 F F F 16 320-339 18 18 18 37 37 37 F F F 17 340-359 18 18 18 38 38 38 F F F 18 360-379 18 18 18 38 38 38 T T T 19 380-399 18 18 18 37 37 37 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 16 16 16 35 35 35 T T T 22 440-459 16 16 16 35 35 35 T T T 23 460-479 16 16 16 35 35 35 F F F 24 480-499 16 16 16 35 35 35 F F F 25 500-519 15 15 15 32 32 32 F F F 26 520-539 16 16 16 33 33 33 F F F 27 540-559 16 16 16 33 33 33 T T T 28 560-579 14 14 14 30 30 30 T T T 29 580-599 12 12 12 23 23 23 F F F Max gen: 18, Max flush: 38 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 19 19 19 39 39 39 F F F 1 020-039 17 17 17 36 36 36 F F F 2 040-059 17 17 17 36 36 36 F F F 3 060-079 14 14 14 28 28 28 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 18 18 18 38 38 38 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 14 14 14 26 26 26 F F F 9 180-199 18 18 18 38 38 38 F F F 10 200-219 19 19 19 39 39 39 F F F 11 220-239 19 19 19 39 39 39 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 18 18 18 38 38 38 F F F 15 300-319 18 18 18 37 37 37 F F F 16 320-339 18 18 18 37 37 37 F F F 17 340-359 18 18 18 38 38 38 F F F 18 360-379 19 19 19 39 39 39 F F F 19 380-399 18 18 18 37 37 37 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 19 19 19 39 39 39 F F F 22 440-459 19 19 19 39 39 39 F F F 23 460-479 16 16 16 35 35 35 F F F 24 480-499 16 16 16 35 35 35 F F F 25 500-519 15 15 15 32 32 32 F F F 26 520-539 16 16 16 33 33 33 F F F 27 540-559 19 19 19 39 39 39 F F F 28 560-579 19 19 19 39 39 39 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 19, Max flush: 39 Begin loop 10 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.793600596Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.793923984Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.793945182Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265} {"msg":"Crucible eaf40e42-22f6-460e-b74c-ff4645b6cd1d has session id: 31de22c9-fd54-4508-93c9-2ba62ea16712","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.79426801Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Upstairs opts: Upstairs UUID: eaf40e42-22f6-460e-b74c-ff4645b6cd1d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794316285Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Crucible stats registered with UUID: eaf40e42-22f6-460e-b74c-ff4645b6cd1d","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794344482Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794572669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265} {"msg":"eaf40e42-22f6-460e-b74c-ff4645b6cd1d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794695677Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794724564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"0","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794741883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"0","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794756141Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.7947721Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794791208Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"2","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794811286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"2","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794825334Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"io task","client":"0","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794839293Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"io task","client":"2","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.794853101Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"io task","client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:52.795051302Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"io task","client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"ds_connection connected from Ok(127.0.0.1:46225)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.795090208Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"io task","client":"2","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:52.795117625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"io task","client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"ds_connection connected from Ok(127.0.0.1:56860)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.795252022Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"io task","client":"0","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:26:52.795279509Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.795294008Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.795310346Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.795403257Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"io task","client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.795718186Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"2","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.795745403Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"2","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.795847843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"0","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.795914966Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"0","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.796304658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"2","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.796432385Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"0","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.796862622Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"2","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.79688537Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"2","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.796899409Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.796955733Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"0","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.796973671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"0","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:26:52.79698752Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.795807822Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"io task","client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"ds_connection connected from Ok(127.0.0.1:65524)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.795929Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"io task","client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.796813093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.796834221Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.797406804Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.797996865Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798027982Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[0]R flush_numbers[0..12]: [39, 36, 36, 28, 17, 17, 38, 36, 26, 38, 39, 39]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.79804688Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[0]R generation[0..12]: [19, 17, 17, 14, 10, 10, 18, 17, 14, 18, 19, 19]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798065029Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798082977Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[1]R flush_numbers[0..12]: [39, 36, 36, 28, 17, 17, 38, 36, 26, 38, 39, 39]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798100285Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[1]R generation[0..12]: [19, 17, 17, 14, 10, 10, 18, 17, 14, 18, 19, 19]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798117733Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798135862Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[2]R flush_numbers[0..12]: [39, 36, 36, 28, 17, 17, 38, 36, 26, 38, 39, 39]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.7981531Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[2]R generation[0..12]: [19, 17, 17, 14, 10, 10, 18, 17, 14, 18, 19, 19]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798170288Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798188876Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 20","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798210634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Generation requested: 20 >= found:20","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798224993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Next flush: 40","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798238252Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.79825172Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"":"downstairs","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798265169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798277248Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"0","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798290566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"1","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798303785Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"client":"2","":"downstairs","session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798317264Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798329752Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"eaf40e42-22f6-460e-b74c-ff4645b6cd1d is now active with session: 31de22c9-fd54-4508-93c9-2ba62ea16712","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798342941Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.7983551Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265,"session_id":"31de22c9-fd54-4508-93c9-2ba62ea16712"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:02.798367479Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 424 len 16384 data: 2 2 2 3 02/30 Write block 331 len 40960 data: 4 4 3 3 2 2 2 2 2 2 03/30 Write block 462 len 4096 data: 5 04/30 Write block 191 len 20480 data: 4 4 4 4 3 05/30 Read block 197 len 8192 06/30 Write block 525 len 24576 data: 4 4 4 4 3 2 07/30 Write block 457 len 28672 data: 3 4 4 4 4 6 4 08/30 Read block 250 len 16384 09/30 Read block 283 len 8192 10/30 Read block 137 len 8192 11/30 Write block 14 len 24576 data: 4 4 5 5 4 3 12/30 Write block 499 len 32768 data: 2 2 2 2 2 2 2 2 13/30 Flush 14/30 Write block 254 len 24576 data: 3 3 3 3 3 3 15/30 Read block 255 len 24576 16/30 Read block 546 len 28672 17/30 Read block 311 len 40960 18/30 Read block 298 len 40960 19/30 Read block 578 len 32768 20/30 Write block 451 len 8192 data: 4 3 21/30 Write block 402 len 12288 data: 2 3 4 22/30 Write block 314 len 40960 data: 3 3 4 5 5 6 6 6 5 4 23/30 Write block 507 len 36864 data: 2 3 3 4 5 4 4 4 4 24/30 Read block 222 len 4096 25/30 Write block 441 len 40960 data: 5 5 5 5 5 5 5 4 4 4 26/30 Write block 406 len 20480 data: 4 4 4 5 5 27/30 Write block 204 len 24576 data: 4 5 7 8 7 6 28/30 Write block 341 len 32768 data: 2 2 2 2 2 2 2 2 29/30 Read block 164 len 40960 30/30 Write block 229 len 40960 data: 2 2 3 3 3 4 3 3 3 3 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:17 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 14 Acked 1013 Write 6 Done Done Done false 15 Acked 1014 Read 6 Done Done Done false 16 Acked 1015 Read 7 Done Done Done false 17 Acked 1016 Read 10 Done Done Done false 18 Acked 1017 Read 10 Done Done Done false 19 Acked 1018 Read 8 Done Done Done false 20 Acked 1019 Write 2 Done Done Done false 21 Acked 1020 Write 3 Done Done Done false 22 Acked 1021 Write 10 Done Done Done false 23 Acked 1022 Write 9 Done Done Done false 24 Acked 1023 Read 1 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Write 5 Done Done Done false 27 Acked 1026 Write 6 Done Done Done false 28 Acked 1027 Write 8 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 16 16 16 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1012 1012 1012 Downstairs last five completed: 1012 1011 1010 1009 1008 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:27:02.8477963Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1265} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 20 20 20 40 40 40 F F F 1 020-039 17 17 17 36 36 36 F F F 2 040-059 17 17 17 36 36 36 F F F 3 060-079 14 14 14 28 28 28 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 18 18 18 38 38 38 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 14 14 14 26 26 26 F F F 9 180-199 20 20 20 40 40 40 F F F 10 200-219 19 19 19 39 39 39 T T T 11 220-239 19 19 19 39 39 39 T T T 12 240-259 4 4 4 6 6 6 T T T 13 260-279 6 6 6 8 8 8 F F F 14 280-299 18 18 18 38 38 38 F F F 15 300-319 18 18 18 37 37 37 T T T 16 320-339 20 20 20 40 40 40 T T T 17 340-359 20 20 20 40 40 40 T T T 18 360-379 19 19 19 39 39 39 F F F 19 380-399 18 18 18 37 37 37 F F F 20 400-419 2 2 2 4 4 4 T T T 21 420-439 20 20 20 40 40 40 F F F 22 440-459 20 20 20 40 40 40 T T T 23 460-479 20 20 20 40 40 40 F F F 24 480-499 20 20 20 40 40 40 F F F 25 500-519 20 20 20 40 40 40 T T T 26 520-539 20 20 20 40 40 40 F F F 27 540-559 19 19 19 39 39 39 F F F 28 560-579 19 19 19 39 39 39 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 20, Max flush: 40 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 20 20 20 40 40 40 F F F 1 020-039 17 17 17 36 36 36 F F F 2 040-059 17 17 17 36 36 36 F F F 3 060-079 14 14 14 28 28 28 F F F 4 080-099 10 10 10 17 17 17 F F F 5 100-119 10 10 10 17 17 17 F F F 6 120-139 18 18 18 38 38 38 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 14 14 14 26 26 26 F F F 9 180-199 20 20 20 40 40 40 F F F 10 200-219 21 21 21 41 41 41 F F F 11 220-239 21 21 21 41 41 41 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 6 6 6 8 8 8 F F F 14 280-299 18 18 18 38 38 38 F F F 15 300-319 21 21 21 41 41 41 F F F 16 320-339 21 21 21 41 41 41 F F F 17 340-359 21 21 21 41 41 41 F F F 18 360-379 19 19 19 39 39 39 F F F 19 380-399 18 18 18 37 37 37 F F F 20 400-419 21 21 21 41 41 41 F F F 21 420-439 20 20 20 40 40 40 F F F 22 440-459 21 21 21 41 41 41 F F F 23 460-479 20 20 20 40 40 40 F F F 24 480-499 20 20 20 40 40 40 F F F 25 500-519 21 21 21 41 41 41 F F F 26 520-539 20 20 20 40 40 40 F F F 27 540-559 19 19 19 39 39 39 F F F 28 560-579 19 19 19 39 39 39 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 21, Max flush: 41 Begin loop 11 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.958974581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.959257203Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.959275372Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273} {"msg":"Crucible ff2acced-e144-4549-a7ff-89a36afb7926 has session id: 17d37ee6-e453-45d6-b91d-1c1d3b429da1","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.959533046Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Upstairs opts: Upstairs UUID: ff2acced-e144-4549-a7ff-89a36afb7926, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.959576342Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Crucible stats registered with UUID: ff2acced-e144-4549-a7ff-89a36afb7926","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.959602359Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.95989299Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273} {"msg":"ff2acced-e144-4549-a7ff-89a36afb7926 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.959957764Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.959988651Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"0","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960005279Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"0","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960019368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960032977Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960046395Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"2","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960060004Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"2","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960089541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"io task","client":"2","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960115928Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"io task","client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960142386Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"io task","client":"0","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ds_connection connected from Ok(127.0.0.1:48809)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960375632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"io task","client":"2","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ds_connection connected from Ok(127.0.0.1:45097)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.9604963Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"io task","client":"0","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:27:17.960528367Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"io task","client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:27:17.960556834Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"io task","client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:27:17.960591551Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960615509Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960637406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.960760344Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"io task","client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.961133508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"2","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.961160705Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"2","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.961175393Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"0","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.961189202Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"0","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.961781673Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"0","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.961940588Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"2","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.962296422Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"0","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.962332549Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"0","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.962357146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.962460426Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"2","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.962478834Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"2","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:17.962495682Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.962013715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"io task","client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ds_connection connected from Ok(127.0.0.1:52818)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.962179428Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"io task","client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.963503137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.963529865Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964192309Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964729435Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964754103Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[0]R flush_numbers[0..12]: [40, 36, 36, 28, 17, 17, 38, 36, 26, 40, 41, 41]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964772971Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[0]R generation[0..12]: [20, 17, 17, 14, 10, 10, 18, 17, 14, 20, 21, 21]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964790969Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964808988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[1]R flush_numbers[0..12]: [40, 36, 36, 28, 17, 17, 38, 36, 26, 40, 41, 41]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964826486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[1]R generation[0..12]: [20, 17, 17, 14, 10, 10, 18, 17, 14, 20, 21, 21]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964843814Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964869292Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[2]R flush_numbers[0..12]: [40, 36, 36, 28, 17, 17, 38, 36, 26, 40, 41, 41]","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2024-07-09T21:27:27.964906058Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[2]R generation[0..12]: [20, 17, 17, 14, 10, 10, 18, 17, 14, 20, 21, 21]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964939565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964962252Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.964975871Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.9649891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Next flush: 42","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.965011318Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.965035165Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"":"downstairs","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.965059013Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.965079201Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"0","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.965103698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"1","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.965126126Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"client":"2","":"downstairs","session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.965140475Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.965161213Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"ff2acced-e144-4549-a7ff-89a36afb7926 is now active with session: 17d37ee6-e453-45d6-b91d-1c1d3b429da1","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.96518503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.965207568Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273,"session_id":"17d37ee6-e453-45d6-b91d-1c1d3b429da1"} {"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:27.965223717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 568 len 40960 data: 3 2 2 4 4 4 4 5 6 5 02/30 Write block 93 len 28672 data: 3 4 4 5 5 4 4 03/30 Write block 273 len 4096 data: 3 04/30 Write block 391 len 4096 data: 6 05/30 Read block 302 len 36864 06/30 Write block 69 len 24576 data: 3 3 3 3 3 3 07/30 Write block 91 len 40960 data: 3 3 4 5 5 6 6 5 5 4 08/30 Write block 123 len 20480 data: 2 2 2 3 3 09/30 Write block 86 len 16384 data: 3 2 3 3 10/30 Read block 400 len 16384 11/30 Read block 585 len 36864 12/30 Read block 496 len 28672 13/30 Flush 14/30 Write block 356 len 20480 data: 4 4 4 3 3 15/30 Read block 453 len 24576 16/30 Write block 294 len 32768 data: 3 4 5 5 5 4 4 3 17/30 Write block 172 len 4096 data: 3 18/30 Read block 525 len 28672 19/30 Write block 83 len 36864 data: 4 4 3 4 3 4 4 3 4 20/30 Flush 21/30 Write block 535 len 12288 data: 2 2 2 22/30 Flush 23/30 Write block 191 len 40960 data: 5 5 5 5 4 2 3 3 4 4 24/30 Read block 380 len 12288 25/30 Flush 26/30 Write block 215 len 12288 data: 2 2 2 27/30 Read block 574 len 16384 28/30 Write block 71 len 32768 data: 4 4 4 4 2 2 2 3 29/30 Write block 58 len 20480 data: 3 3 2 2 2 30/30 Write block 300 len 32768 data: 5 4 2 2 2 2 3 3 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Write 8 Done Done Done false 29 Acked 1028 Write 5 Done Done Done false 30 Acked 1029 Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:27:28.094361678Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1273} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 20 20 20 40 40 40 F F F 1 020-039 17 17 17 36 36 36 F F F 2 040-059 17 17 17 36 36 36 T T T 3 060-079 22 22 22 42 42 42 T T T 4 080-099 22 22 22 43 43 43 F F F 5 100-119 22 22 22 42 42 42 F F F 6 120-139 22 22 22 42 42 42 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 22 22 22 45 45 45 F F F 10 200-219 22 22 22 45 45 45 T T T 11 220-239 21 21 21 41 41 41 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 22 22 22 43 43 43 F F F 15 300-319 22 22 22 43 43 43 T T T 16 320-339 21 21 21 41 41 41 F F F 17 340-359 22 22 22 43 43 43 F F F 18 360-379 22 22 22 43 43 43 F F F 19 380-399 22 22 22 42 42 42 F F F 20 400-419 21 21 21 41 41 41 F F F 21 420-439 20 20 20 40 40 40 F F F 22 440-459 21 21 21 41 41 41 F F F 23 460-479 20 20 20 40 40 40 F F F 24 480-499 20 20 20 40 40 40 F F F 25 500-519 21 21 21 41 41 41 F F F 26 520-539 22 22 22 44 44 44 F F F 27 540-559 19 19 19 39 39 39 F F F 28 560-579 22 22 22 42 42 42 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 22, Max flush: 45 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 20 20 20 40 40 40 F F F 1 020-039 17 17 17 36 36 36 F F F 2 040-059 23 23 23 46 46 46 F F F 3 060-079 23 23 23 46 46 46 F F F 4 080-099 22 22 22 43 43 43 F F F 5 100-119 22 22 22 42 42 42 F F F 6 120-139 22 22 22 42 42 42 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 22 22 22 45 45 45 F F F 10 200-219 23 23 23 46 46 46 F F F 11 220-239 21 21 21 41 41 41 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 22 22 22 43 43 43 F F F 15 300-319 23 23 23 46 46 46 F F F 16 320-339 21 21 21 41 41 41 F F F 17 340-359 22 22 22 43 43 43 F F F 18 360-379 22 22 22 43 43 43 F F F 19 380-399 22 22 22 42 42 42 F F F 20 400-419 21 21 21 41 41 41 F F F 21 420-439 20 20 20 40 40 40 F F F 22 440-459 21 21 21 41 41 41 F F F 23 460-479 20 20 20 40 40 40 F F F 24 480-499 20 20 20 40 40 40 F F F 25 500-519 21 21 21 41 41 41 F F F 26 520-539 22 22 22 44 44 44 F F F 27 540-559 19 19 19 39 39 39 F F F 28 560-579 22 22 22 42 42 42 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 23, Max flush: 46 Begin loop 12 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.643910335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.644300756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.644328624Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281} {"msg":"Crucible 2b1008ea-d9f6-4867-8564-a58f82026836 has session id: 5fa5c156-3fce-4ced-91dc-7745a073cbe2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.644704636Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Upstairs opts: Upstairs UUID: 2b1008ea-d9f6-4867-8564-a58f82026836, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.644780869Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Crucible stats registered with UUID: 2b1008ea-d9f6-4867-8564-a58f82026836","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.644808906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645086938Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281} {"msg":"2b1008ea-d9f6-4867-8564-a58f82026836 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645184419Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645210056Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645226145Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645240443Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"1","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645254352Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"1","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.64526778Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"2","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645281519Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"2","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645294778Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"io task","client":"2","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645308756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"io task","client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645322625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"io task","client":"1","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"ds_connection connected from Ok(127.0.0.1:53595)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645564061Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"io task","client":"2","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"ds_connection connected from Ok(127.0.0.1:65223)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.645866011Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"io task","client":"1","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:27:42.645904188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"io task","client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:27:42.645932245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"io task","client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:27:42.645997818Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.646024206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.646039974Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.646145584Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"io task","client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.646332775Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"2","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.646355783Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"2","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.646370631Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"1","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.646389469Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"1","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.646945384Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"2","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.647081511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"1","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.647436406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"2","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.647463013Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"2","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.647478392Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.647538586Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"1","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.647557574Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"1","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:42.647574412Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.647603065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"io task","client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"ds_connection connected from Ok(127.0.0.1:57687)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.647912465Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"io task","client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.648753221Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.648785868Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649309076Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649842653Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649865181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[0]R flush_numbers[0..12]: [40, 36, 46, 46, 43, 42, 42, 36, 43, 45, 46, 41]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649880039Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[0]R generation[0..12]: [20, 17, 23, 23, 22, 22, 22, 17, 22, 22, 23, 21]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649894458Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649908907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[1]R flush_numbers[0..12]: [40, 36, 46, 46, 43, 42, 42, 36, 43, 45, 46, 41]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649922925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[1]R generation[0..12]: [20, 17, 23, 23, 22, 22, 22, 17, 22, 22, 23, 21]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649936624Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649950792Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[2]R flush_numbers[0..12]: [40, 36, 46, 46, 43, 42, 42, 36, 43, 45, 46, 41]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649964481Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[2]R generation[0..12]: [20, 17, 23, 23, 22, 22, 22, 17, 22, 22, 23, 21]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649982469Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.649997098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Max found gen is 24"Wait for a query_work_queue command to finish before sending IO ,"v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650013966Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650027705Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Next flush: 47","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650040994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650054162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"":"downstairs","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650067401Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.65007927Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"0","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650092538Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"1","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650108477Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"client":"2","":"downstairs","session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650132824Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650154732Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"2b1008ea-d9f6-4867-8564-a58f82026836 is now active with session: 5fa5c156-3fce-4ced-91dc-7745a073cbe2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.6501781Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650200018Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281,"session_id":"5fa5c156-3fce-4ced-91dc-7745a073cbe2"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:27:52.650218346Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 117 len 40960 data: 2 2 2 2 2 2 3 3 3 4 03/30 Read block 181 len 40960 04/30 Write block 515 len 28672 data: 5 3 3 2 2 2 2 05/30 Flush 06/30 Write block 24 len 40960 data: 3 4 4 4 5 5 4 4 4 4 07/30 Flush 08/30 Write block 341 len 40960 data: 3 3 3 3 3 3 3 3 2 2 09/30 Read block 297 len 32768 10/30 Write block 222 len 4096 data: 3 11/30 Write block 458 len 12288 data: 5 5 5 12/30 Flush 13/30 Write block 221 len 20480 data: 3 4 3 3 2 14/30 Read block 18 len 20480 15/30 Write block 427 len 32768 data: 4 3 3 3 3 3 4 4 16/30 Read block 117 len 28672 17/30 Read block 439 len 12288 18/30 Write block 97 len 28672 data: 7 6 6 5 4 4 5 19/30 Read block 122 len 40960 20/30 Read block 125 len 28672 21/30 Write block 213 len 12288 data: 3 2 3 22/30 Read block 346 len 8192 23/30 Write block 27 len 32768 data: 5 6 6 5 5 5 5 4 24/30 Write block 374 len 32768 data: 3 3 2 2 2 3 3 3 25/30 Write block 388 len 40960 data: 6 6 5 7 6 6 6 5 3 2 26/30 Read block 36 len 40960 27/30 Read block 157 len 24576 28/30 Read block 72 len 20480 29/30 Write block 533 len 8192 data: 2 2 30/30 Write block 307 len 32768 data: 4 3 3 3 3 3 3 4 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:18 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 13 Acked 1012 Write 5 Done Done Done false 14 Acked 1013 Read 5 Done Done Done false 15 Acked 1014 Write 8 Done Done Done false 16 Acked 1015 Read 7 Done Done Done false 17 Acked 1016 Read 3 Done Done Done false 18 Acked 1017 Write 7 Done Done Done false 19 Acked 1018 Read 10 Done Done Done false 20 Acked 1019 Read 7 Done Done Done false 21 Acked 1020 Write 3 Done Done Done false 22 Acked 1021 Read 2 Done Done Done false 23 Acked 1022 Write 8 Done Done Done false 24 Acked 1023 Write 8 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Read 6 Done Done Done false 28 Acked 1027 Read 5 Done Done Done false 29 Acked 1028 Write 2 Done Done Done false 30 Acked 1029 Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 17 17 17 51 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1011 1011 1011 Downstairs last five completed: 1011 1010 1009 1008 1007 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:27:52.713327571Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1281} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 20 20 20 40 40 40 F F F 1 020-039 24 24 24 49 49 49 T T T 2 040-059 23 23 23 46 46 46 F F F 3 060-079 23 23 23 46 46 46 F F F 4 080-099 22 22 22 43 43 43 T T T 5 100-119 24 24 24 48 48 48 T T T 6 120-139 24 24 24 48 48 48 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 22 22 22 45 45 45 F F F 10 200-219 23 23 23 46 46 46 T T T 11 220-239 24 24 24 50 50 50 T T T 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 22 22 22 43 43 43 F F F 15 300-319 23 23 23 46 46 46 T T T 16 320-339 21 21 21 41 41 41 F F F 17 340-359 24 24 24 50 50 50 F F F 18 360-379 22 22 22 43 43 43 T T T 19 380-399 22 22 22 42 42 42 T T T 20 400-419 21 21 21 41 41 41 F F F 21 420-439 20 20 20 40 40 40 T T T 22 440-459 24 24 24 50 50 50 F F F 23 460-479 24 24 24 50 50 50 F F F 24 480-499 20 20 20 40 40 40 F F F 25 500-519 24 24 24 48 48 48 F F F 26 520-539 24 24 24 48 48 48 T T T 27 540-559 19 19 19 39 39 39 F F F 28 560-579 22 22 22 42 42 42 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 24, Max flush: 50 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 20 20 20 40 40 40 F F F 1 020-039 25 25 25 51 51 51 F F F 2 040-059 23 23 23 46 46 46 F F F 3 060-079 23 23 23 46 46 46 F F F 4 080-099 25 25 25 51 51 51 F F F 5 100-119 25 25 25 51 51 51 F F F 6 120-139 24 24 24 48 48 48 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 22 22 22 45 45 45 F F F 10 200-219 25 25 25 51 51 51 F F F 11 220-239 25 25 25 51 51 51 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 22 22 22 43 43 43 F F F 15 300-319 25 25 25 51 51 51 F F F 16 320-339 21 21 21 41 41 41 F F F 17 340-359 24 24 24 50 50 50 F F F 18 360-379 25 25 25 51 51 51 F F F 19 380-399 25 25 25 51 51 51 F F F 20 400-419 21 21 21 41 41 41 F F F 21 420-439 25 25 25 51 51 51 F F F 22 440-459 24 24 24 50 50 50 F F F 23 460-479 24 24 24 50 50 50 F F F 24 480-499 20 20 20 40 40 40 F F F 25 500-519 24 24 24 48 48 48 F F F 26 520-539 25 25 25 51 51 51 F F F 27 540-559 19 19 19 39 39 39 F F F 28 560-579 22 22 22 42 42 42 F F F 29 580-599 12 12 12 23 23 23 F F F Max gen: 25, Max flush: 51 Begin loop 13 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.557640885Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.557927727Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.557948725Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289} {"msg":"Crucible 87ca705d-f108-4830-8996-fd9808f8d7ea has session id: a505e8a7-f9a7-41a3-9ff4-c3518efeda5e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.55820368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Upstairs opts: Upstairs UUID: 87ca705d-f108-4830-8996-fd9808f8d7ea, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558249165Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Crucible stats registered with UUID: 87ca705d-f108-4830-8996-fd9808f8d7ea","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558273893Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558631717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289} {"msg":"87ca705d-f108-4830-8996-fd9808f8d7ea active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.55869792Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558715369Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558730357Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558744106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"1","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558758185Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"1","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558771993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"2","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558785672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"2","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558799001Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"io task","client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558813099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"io task","client":"2","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.558826838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"io task","client":"1","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:07.558986652Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"io task","client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:59548)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.559051596Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"io task","client":"1","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:07.559087052Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"io task","client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:40199)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.559114809Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"io task","client":"2","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:07.559247066Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.559270394Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.559284502Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.559418459Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"io task","client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.559928889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"2","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.559967505Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"2","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.559996922Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"1","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.560024179Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"1","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.560608001Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"2","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.560749437Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"1","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.561059557Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"2","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.561098842Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"2","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.561114941Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.561232089Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"1","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.561251897Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"1","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:07.561265906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.560569522Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"io task","client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:52784)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.56068732Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"io task","client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.562137646Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.562169253Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.562730328Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563245786Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563268794Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[0]R flush_numbers[0..12]: [40, 51, 46, 46, 51, 51, 48, 36, 43, 45, 51, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563284643Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[0]R generation[0..12]: [20, 25, 23, 23, 25, 25, 24, 17, 22, 22, 25, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563298951Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.56331371Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[1]R flush_numbers[0..12]: [40, 51, 46, 46, 51, 51, 48, 36, 43, 45, 51, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563327938Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[1]R generation[0..12]: [20, 25, 23, 23, 25, 25, 24, 17, 22, 22, 25, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563341817Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563359715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[2]R flush_numbers[0..12]: [40, 51, 46, 46, 51, 51, 48, 36, 43, 45, 51, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563377073Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[2]R generation[0..12]: [20, 25, 23, 23, 25, 25, 24, 17, 22, 22, 25, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563401651Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563428588Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563453696Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563478743Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Next flush: 52","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563503271Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563519909Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"":"downstairs","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563535698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563556676Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"0","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563581643Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"1","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563602751Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"client":"2","":"downstairs","session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563627009Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563650586Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"87ca705d-f108-4830-8996-fd9808f8d7ea is now active with session: a505e8a7-f9a7-41a3-9ff4-c3518efeda5e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563666775Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563682343Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289,"session_id":"a505e8a7-f9a7-41a3-9ff4-c3518efeda5e"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:17.563703921Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 305 len 8192 data: 3 4 02/30 Read block 275 len 40960 03/30 Read block 503 len 20480 04/30 Flush 05/30 Write block 308 len 32768 data: 4 4 4 4 4 4 5 4 06/30 Write block 490 len 8192 data: 5 4 07/30 Write block 589 len 32768 data: 2 2 2 2 2 3 3 3 08/30 Read block 442 len 8192 09/30 Read block 0 len 16384 10/30 Read block 420 len 8192 11/30 Read block 152 len 16384 12/30 Read block 545 len 40960 13/30 Write block 405 len 24576 data: 3 5 5 5 6 6 14/30 Flush 15/30 Write block 342 len 24576 data: 4 4 4 4 4 4 16/30 Read block 318 len 20480 17/30 Write block 80 len 12288 data: 3 3 4 18/30 Read block 6 len 36864 19/30 Read block 408 len 4096 20/30 Read block 88 len 12288 21/30 Read block 513 len 8192 22/30 Read block 298 len 16384 23/30 Read block 449 len 12288 24/30 Read block 64 len 36864 25/30 Write block 411 len 8192 data: 3 3 26/30 Read block 191 len 12288 27/30 Write block 114 len 8192 data: 2 2 28/30 Write block 347 len 4096 data: 5 29/30 Write block 408 len 4096 data: 6 30/30 Read block 149 len 32768 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Read 9 Done Done Done false 20 Acked 1019 Read 1 Done Done Done false 21 Acked 1020 Read 3 Done Done Done false 22 Acked 1021 Read 2 Done Done Done false 23 Acked 1022 Read 4 Done Done Done false 24 Acked 1023 Read 3 Done Done Done false 25 Acked 1024 Read 9 Done Done Done false 26 Acked 1025 Write 2 Done Done Done false 27 Acked 1026 Read 3 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Write 1 Done Done Sent false 30 Acked 1029 Write 1 Done Done Sent false 31 Acked 1030 Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 3 4 Done 13 12 10 35 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:17.611866408Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1289} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 20 20 20 40 40 40 F F F 1 020-039 25 25 25 51 51 51 F F F 2 040-059 23 23 23 46 46 46 F F F 3 060-079 23 23 23 46 46 46 F F F 4 080-099 26 26 26 54 54 54 F F F 5 100-119 25 25 25 51 51 51 T T T 6 120-139 24 24 24 48 48 48 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 22 22 22 45 45 45 F F F 10 200-219 25 25 25 51 51 51 F F F 11 220-239 25 25 25 51 51 51 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 22 22 22 43 43 43 F F F 15 300-319 26 26 26 53 53 53 F F F 16 320-339 21 21 21 41 41 41 F F F 17 340-359 26 26 26 54 54 54 T T T 18 360-379 25 25 25 51 51 51 F F F 19 380-399 25 25 25 51 51 51 F F F 20 400-419 26 26 26 53 53 53 T T T 21 420-439 25 25 25 51 51 51 F F F 22 440-459 24 24 24 50 50 50 F F F 23 460-479 24 24 24 50 50 50 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 24 24 24 48 48 48 F F F 26 520-539 25 25 25 51 51 51 F F F 27 540-559 19 19 19 39 39 39 F F F 28 560-579 22 22 22 42 42 42 F F F 29 580-599 26 26 26 53 53 53 F F F Max gen: 26, Max flush: 54 A Difference in extent metadata is expected here On loop 13 Verifying data now No change after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 20 20 20 40 40 40 F F F 1 020-039 25 25 25 51 51 51 F F F 2 040-059 23 23 23 46 46 46 F F F 3 060-079 23 23 23 46 46 46 F F F 4 080-099 26 26 26 54 54 54 F F F 5 100-119 27 27 27 55 55 55 F F F 6 120-139 24 24 24 48 48 48 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 22 22 22 45 45 45 F F F 10 200-219 25 25 25 51 51 51 F F F 11 220-239 25 25 25 51 51 51 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 22 22 22 43 43 43 F F F 15 300-319 26 26 26 53 53 53 F F F 16 320-339 21 21 21 41 41 41 F F F 17 340-359 27 27 27 55 55 55 F F F 18 360-379 25 25 25 51 51 51 F F F 19 380-399 25 25 25 51 51 51 F F F 20 400-419 27 27 27 55 55 55 F F F 21 420-439 25 25 25 51 51 51 F F F 22 440-459 24 24 24 50 50 50 F F F 23 460-479 24 24 24 50 50 50 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 24 24 24 48 48 48 F F F 26 520-539 25 25 25 51 51 51 F F F 27 540-559 19 19 19 39 39 39 F F F 28 560-579 22 22 22 42 42 42 F F F 29 580-599 26 26 26 53 53 53 F F F Max gen: 27, Max flush: 55 Begin loop 14 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.965461796Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966035049Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966065086Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297} {"msg":"Crucible dd4eb9ca-e3b9-46ae-a042-e9675e1ba325 has session id: 3f9284cd-234a-4916-a8c8-043bc48e65ab","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966093944Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Upstairs opts: Upstairs UUID: dd4eb9ca-e3b9-46ae-a042-e9675e1ba325, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966133649Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Crucible stats registered with UUID: dd4eb9ca-e3b9-46ae-a042-e9675e1ba325","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966160567Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966526561Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297} {"msg":"dd4eb9ca-e3b9-46ae-a042-e9675e1ba325 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966556148Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966578576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966604673Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.96662862Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"1","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966654118Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"1","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966676636Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"2","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966698444Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"2","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966722451Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"io task","client":"1","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966747809Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"io task","client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.966771096Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"io task","client":"2","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:31.967003393Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"io task","client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:31.967043339Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"io task","client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:31.967070127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:34818)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.967090135Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"io task","client":"2","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.967113132Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.967134201Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.967155318Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"io task","client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:36583)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.967178956Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"io task","client":"1","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.967655159Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"2","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.967682366Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"2","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.967700174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"1","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.967724772Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"1","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.968316263Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"2","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.968353289Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"1","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.968832172Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"1","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.968857819Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"1","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.968886217Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.968904935Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"2","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.968919024Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"2","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:31.968932502Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.968510749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"io task","client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:37508)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.968940717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"io task","client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.970021019Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.970047747Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.970530169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971067306Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971090824Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[0]R flush_numbers[0..12]: [40, 51, 46, 46, 54, 55, 48, 36, 43, 45, 51, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971108842Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[0]R generation[0..12]: [20, 25, 23, 23, 26, 27, 24, 17, 22, 22, 25, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.97112629Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971144198Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[1]R flush_numbers[0..12]: [40, 51, 46, 46, 54, 55, 48, 36, 43, 45, 51, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971161566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[1]R generation[0..12]: [20, 25, 23, 23, 26, 27, 24, 17, 22, 22, 25, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971183594Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971201822Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[2]R flush_numbers[0..12]: [40, 51, 46, 46, 54, 55, 48, 36, 43, 45, 51, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971219251Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R generation[0..12]: [20, 25, 23, 23, 26, 27, 24, 17, 22, 22, 25, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971238789Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971257347Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Max found gen is 28","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971273885Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Generation requested: 28 >= found:28","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971290664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Next flush: 56","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971306892Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971323341Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"":"downstairs","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971339819Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971355597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"0","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971370216Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"1","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971383215Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"client":"2","":"downstairs","session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971396593Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971409062Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"dd4eb9ca-e3b9-46ae-a042-e9675e1ba325 is now active with session: 3f9284cd-234a-4916-a8c8-043bc48e65ab","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.97142418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971436479Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297,"session_id":"3f9284cd-234a-4916-a8c8-043bc48e65ab"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:41.971448618Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 2 len 16384 data: 2 2 2 2 02/30 Flush 03/30 Write block 420 len 32768 data: 2 2 2 2 3 3 3 5 04/30 Write block 468 len 24576 data: 4 4 4 3 3 3 05/30 Read block 287 len 20480 06/30 Read block 184 len 36864 07/30 Read block 293 len 40960 08/30 Write block 337 len 32768 data: 3 3 3 3 4 5 5 5 09/30 Read block 252 len 20480 10/30 Read block 186 len 28672 11/30 Read block 507 len 36864 12/30 Flush 13/30 Write block 280 len 24576 data: 2 2 2 2 3 3 14/30 Write block 92 len 8192 data: 4 5 15/30 Write block 440 len 36864 data: 5 6 6 6 6 6 6 6 5 16/30 Read block 35 len 40960 17/30 Write block 366 len 4096 data: 2 18/30 Read block 589 len 24576 19/30 Write block 326 len 32768 data: 6 6 6 5 5 5 5 4 20/30 Read block 528 len 40960 21/30 Write block 210 len 40960 data: 5 5 4 4 3 4 3 3 2 3 22/30 Flush 23/30 Write block 190 len 28672 data: 4 6 6 6 6 5 3 24/30 Write block 356 len 40960 data: 5 5 5 4 4 3 3 2 2 2 25/30 Read block 299 len 28672 26/30 Write block 412 len 8192 data: 4 2 27/30 Write block 594 len 24576 data: 4 4 4 3 3 3 28/30 Write block 546 len 4096 data: 4 29/30 Write block 513 len 8192 data: 5 5 30/30 Write block 551 len 16384 data: 5 4 3 3 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Read 6 Done Done Done false 20 Acked 1019 Write 8 Done Done Done false 21 Acked 1020 Read 10 Done Done Done false 22 Acked 1021 Write 10 Done Done Done false 23 Acked 1022 Flush 0 Done Done Sent false 24 Acked 1023 Write 7 Done Done Sent false 25 Acked 1024 Write 10 Done Done Sent false 26 Acked 1025 Read 7 Done Done Sent false 27 Acked 1026 Write 2 Done Done Sent false 28 Acked 1027 Write 6 Done Done Sent false 29 Acked 1028 Write 1 Done Done Sent false 30 Acked 1029 Write 2 Done Done Sent false 31 Acked 1030 Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 9 11 Done 12 12 4 28 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:42.024687581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1297} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 F F F 1 020-039 25 25 25 51 51 51 F F F 2 040-059 23 23 23 46 46 46 F F F 3 060-079 23 23 23 46 46 46 F F F 4 080-099 28 28 28 58 58 58 F F F 5 100-119 27 27 27 55 55 55 F F F 6 120-139 24 24 24 48 48 48 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 22 22 22 45 45 45 T T T 10 200-219 28 28 28 59 59 59 F F F 11 220-239 25 25 25 51 51 51 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 28 28 28 58 58 58 F F F 15 300-319 26 26 26 53 53 53 F F F 16 320-339 28 28 28 59 59 59 F F F 17 340-359 28 28 28 57 57 57 T T T 18 360-379 28 28 28 58 58 58 T T T 19 380-399 25 25 25 51 51 51 F F F 20 400-419 27 27 27 55 55 55 T T F <--- 21 420-439 28 28 28 57 57 57 F F F 22 440-459 28 28 28 58 58 58 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 24 24 24 48 48 48 T T F <--- 26 520-539 25 25 25 51 51 51 F F F 27 540-559 19 19 19 39 39 39 T T F <--- 28 560-579 22 22 22 42 42 42 F F F 29 580-599 26 26 26 53 53 53 T T F <--- Max gen: 28, Max flush: 59 Error: Difference in extent metadata found! A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 F F F 1 020-039 25 25 25 51 51 51 F F F 2 040-059 23 23 23 46 46 46 F F F 3 060-079 23 23 23 46 46 46 F F F 4 080-099 28 28 28 58 58 58 F F F 5 100-119 27 27 27 55 55 55 F F F 6 120-139 24 24 24 48 48 48 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 29 29 29 60 60 60 F F F 10 200-219 28 28 28 59 59 59 F F F 11 220-239 25 25 25 51 51 51 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 28 28 28 58 58 58 F F F 15 300-319 26 26 26 53 53 53 F F F 16 320-339 28 28 28 59 59 59 F F F 17 340-359 29 29 29 60 60 60 F F F 18 360-379 29 29 29 60 60 60 F F F 19 380-399 25 25 25 51 51 51 F F F 20 400-419 29 29 29 60 60 60 F F F 21 420-439 28 28 28 57 57 57 F F F 22 440-459 28 28 28 58 58 58 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 29 29 29 60 60 60 F F F 26 520-539 25 25 25 51 51 51 F F F 27 540-559 29 29 29 60 60 60 F F F 28 560-579 22 22 22 42 42 42 F F F 29 580-599 29 29 29 60 60 60 F F F Max gen: 29, Max flush: 60 Begin loop 15 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.689318984Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.689604545Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.689626333Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305} {"msg":"Crucible 6d20252b-1d66-4137-998a-80e754921d64 has session id: 626f4675-8fc0-4d12-b0b7-61cd16eac069","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690116105Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Upstairs opts: Upstairs UUID: 6d20252b-1d66-4137-998a-80e754921d64, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.69015996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Crucible stats registered with UUID: 6d20252b-1d66-4137-998a-80e754921d64","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690183088Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690456901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305} {"msg":"6d20252b-1d66-4137-998a-80e754921d64 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690521864Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690547512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"0","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.69056383Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"0","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690578029Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"1","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690595127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"1","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690619235Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690637753Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690651422Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"io task","client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.69066554Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"io task","client":"0","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690679499Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"io task","client":"1","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:52.690819205Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"io task","client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:52.690857501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"io task","client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:28:52.690885059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.690907896Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.691026794Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"ds_connection connected from Ok(127.0.0.1:65512)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.691055962Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"io task","client":"1","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"ds_connection connected from Ok(127.0.0.1:43428)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.691163071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"io task","client":"0","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.691196158Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"io task","client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.691663881Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"0","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.691697718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"0","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.691724645Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"1","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.691786679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"1","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.692355443Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"1","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.69238267Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"0","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.692862563Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"0","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.692892469Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"0","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.692908108Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.692936315Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"1","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.692951044Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"1","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:28:52.692964482Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.692437611Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"io task","client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"ds_connection connected from Ok(127.0.0.1:49162)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.692619933Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"io task","client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.693646451Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.693673569Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694191917Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694722615Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694746102Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[0]R flush_numbers[0..12]: [56, 51, 46, 46, 58, 55, 48, 36, 43, 60, 59, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694761761Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[0]R generation[0..12]: [28, 25, 23, 23, 28, 27, 24, 17, 22, 29, 28, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.69477639Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694790888Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[1]R flush_numbers[0..12]: [56, 51, 46, 46, 58, 55, 48, 36, 43, 60, 59, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694808696Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[1]R generation[0..12]: [28, 25, 23, 23, 28, 27, 24, 17, 22, 29, 28, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694822685Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694837004Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[2]R flush_numbers[0..12]: [56, 51, 46, 46, 58, 55, 48, 36, 43, 60, 59, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694850882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[2]R generation[0..12]: [28, 25, 23, 23, 28, 27, 24, 17, 22, 29, 28, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694864751Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694878909Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694892068Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694905607Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Next flush: 61","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694922325Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694935454Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"":"downstairs","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694948872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694960821Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"0","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.69497426Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"1","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.694987549Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"client":"2","":"downstairs","session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.695005337Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.695028125Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"6d20252b-1d66-4137-998a-80e754921d64 is now active with session: 626f4675-8fc0-4d12-b0b7-61cd16eac069","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.695042663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.695054972Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305,"session_id":"626f4675-8fc0-4d12-b0b7-61cd16eac069"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:02.695067251Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 35 len 24576 data: 4 4 5 4 4 3 02/30 Write block 286 len 28672 data: 3 3 4 4 2 2 2 03/30 Read block 432 len 32768 04/30 Write block 43 len 36864 data: 3 2 2 2 2 2 2 2 2 05/30 Read block 359 len 36864 06/30 Read block 442 len 24576 07/30 Flush 08/30 Write block 347 len 28672 data: 6 4 3 3 2 2 4 09/30 Write block 323 len 12288 data: 5 5 6 10/30 Read block 83 len 32768 11/30 Read block 183 len 4096 12/30 Read block 392 len 20480 13/30 Read block 203 len 8192 14/30 Read block 557 len 12288 15/30 Read block 34 len 28672 16/30 Write block 130 len 24576 data: 2 2 4 4 4 4 17/30 Read block 92 len 4096 18/30 Write block 364 len 28672 data: 3 3 3 2 3 3 3 19/30 Read block 586 len 36864 20/30 Read block 428 len 4096 21/30 Read block 239 len 20480 22/30 Read block 509 len 20480 23/30 Read block 310 len 28672 24/30 Write block 399 len 12288 data: 2 2 2 25/30 Write block 409 len 8192 data: 7 7 26/30 Flush 27/30 Write block 389 len 36864 data: 7 6 8 7 7 7 6 4 3 28/30 Write block 593 len 12288 data: 3 5 5 29/30 Read block 385 len 16384 30/30 Read block 378 len 24576 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Write 3 Done Done Done false 30 Acked 1029 Read 4 Done Done Done false 31 Acked 1030 Read 6 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 0 2 Done 3 3 4 10 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:29:02.771592496Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1305} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 F F F 1 020-039 30 30 30 61 61 61 F F F 2 040-059 30 30 30 61 61 61 F F F 3 060-079 23 23 23 46 46 46 F F F 4 080-099 28 28 28 58 58 58 F F F 5 100-119 27 27 27 55 55 55 F F F 6 120-139 30 30 30 62 62 62 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 29 29 29 60 60 60 F F F 10 200-219 28 28 28 59 59 59 F F F 11 220-239 25 25 25 51 51 51 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 30 30 30 61 61 61 F F F 15 300-319 26 26 26 53 53 53 F F F 16 320-339 30 30 30 62 62 62 F F F 17 340-359 30 30 30 62 62 62 F F F 18 360-379 30 30 30 62 62 62 F F F 19 380-399 30 30 30 63 63 63 T T T 20 400-419 30 30 30 63 63 63 F F F 21 420-439 28 28 28 57 57 57 F F F 22 440-459 28 28 28 58 58 58 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 29 29 29 60 60 60 F F F 26 520-539 25 25 25 51 51 51 F F F 27 540-559 29 29 29 60 60 60 F F F 28 560-579 22 22 22 42 42 42 F F F 29 580-599 29 29 29 60 60 60 T T T Max gen: 30, Max flush: 63 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 F F F 1 020-039 30 30 30 61 61 61 F F F 2 040-059 30 30 30 61 61 61 F F F 3 060-079 23 23 23 46 46 46 F F F 4 080-099 28 28 28 58 58 58 F F F 5 100-119 27 27 27 55 55 55 F F F 6 120-139 30 30 30 62 62 62 F F F 7 140-159 17 17 17 36 36 36 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 29 29 29 60 60 60 F F F 10 200-219 28 28 28 59 59 59 F F F 11 220-239 25 25 25 51 51 51 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 30 30 30 61 61 61 F F F 15 300-319 26 26 26 53 53 53 F F F 16 320-339 30 30 30 62 62 62 F F F 17 340-359 30 30 30 62 62 62 F F F 18 360-379 30 30 30 62 62 62 F F F 19 380-399 31 31 31 64 64 64 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 28 28 28 57 57 57 F F F 22 440-459 28 28 28 58 58 58 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 29 29 29 60 60 60 F F F 26 520-539 25 25 25 51 51 51 F F F 27 540-559 29 29 29 60 60 60 F F F 28 560-579 22 22 22 42 42 42 F F F 29 580-599 31 31 31 64 64 64 F F F Max gen: 31, Max flush: 64 Begin loop 16 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.079761616Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080145098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080173475Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313} {"msg":"Crucible 2967ef2f-d5b9-41ec-8385-ccdb95e29377 has session id: 06c69afd-b2b3-4a45-a4be-f3cfd361ab37","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080435329Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Upstairs opts: Upstairs UUID: 2967ef2f-d5b9-41ec-8385-ccdb95e29377, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080487734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Crucible stats registered with UUID: 2967ef2f-d5b9-41ec-8385-ccdb95e29377","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080515791Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080764276Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313} {"msg":"2967ef2f-d5b9-41ec-8385-ccdb95e29377 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080854617Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080876675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080903262Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.08092845Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"1","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080951988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"1","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.080975575Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"2","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.081001243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"2","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.08102545Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"io task","client":"2","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.081050888Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"io task","client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.081076005Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"io task","client":"1","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:29:17.081197543Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"io task","client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:29:17.081241649Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"io task","client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"ds_connection connected from Ok(127.0.0.1:49447)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.081270516Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"io task","client":"2","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"ds_connection connected from Ok(127.0.0.1:55743)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.081296763Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"io task","client":"1","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:29:17.081324991Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.081447799Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.081474386Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.081503143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"io task","client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.082083725Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"1","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.082106953Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"1","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.082121682Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"2","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.08213557Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"2","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.082781616Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"1","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.082809473Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"2","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.083330252Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"1","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.083357219Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"1","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.083375608Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.083399635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"2","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.083413714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"2","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:17.083427242Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.082269852Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"io task","client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"ds_connection connected from Ok(127.0.0.1:64762)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.082567673Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"io task","client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.083789382Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.083821119Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.084485613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.084972884Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.084997952Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[0]R flush_numbers[0..12]: [56, 61, 61, 46, 58, 55, 62, 36, 43, 60, 59, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.08501646Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[0]R generation[0..12]: [28, 30, 30, 23, 28, 27, 30, 17, 22, 29, 28, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085034808Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085052986Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[1]R flush_numbers[0..12]: [56, 61, 61, 46, 58, 55, 62, 36, 43, 60, 59, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085070305Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[1]R generation[0..12]: [28, 30, 30, 23, 28, 27, 30, 17, 22, 29, 28, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085091623Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.08511848Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[2]R flush_numbers[0..12]: [56, 61, 61, 46, 58, 55, 62, 36, 43, 60, 59, 51]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:29:27.085148577Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[2]R generation[0..12]: [28, 30, 30, 23, 28, 27, 30, 17, 22, 29, 28, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085179194Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085208841Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085226579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085247697Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Next flush: 65","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085276094Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085306631Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"":"downstairs","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085335698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085352207Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"0","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085380774Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"1","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085407451Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"client":"2","":"downstairs","session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085431599Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085445218Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"2967ef2f-d5b9-41ec-8385-ccdb95e29377 is now active with session: 06c69afd-b2b3-4a45-a4be-f3cfd361ab37","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085464046Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085485483Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313,"session_id":"06c69afd-b2b3-4a45-a4be-f3cfd361ab37"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:27.085508111Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 134 len 8192 data: 5 5 02/30 Read block 596 len 16384 03/30 Write block 178 len 40960 data: 2 3 3 2 2 2 2 2 2 2 04/30 Read block 396 len 32768 05/30 Read block 575 len 32768 06/30 Read block 97 len 16384 07/30 Read block 387 len 24576 08/30 Write block 562 len 4096 data: 6 09/30 Write block 356 len 16384 data: 6 6 6 5 10/30 Flush 11/30 Write block 503 len 40960 data: 3 3 3 3 3 4 4 5 6 5 12/30 Write block 84 len 28672 data: 5 4 5 4 5 5 4 13/30 Read block 328 len 20480 14/30 Read block 151 len 40960 15/30 Write block 158 len 40960 data: 3 2 2 2 2 2 2 2 2 2 16/30 Write block 358 len 24576 data: 7 6 5 4 4 3 17/30 Read block 132 len 24576 18/30 Flush 19/30 Write block 39 len 36864 data: 5 4 2 2 4 3 3 3 3 20/30 Read block 513 len 36864 21/30 Read block 173 len 4096 22/30 Write block 581 len 8192 data: 3 4 23/30 Read block 333 len 40960 24/30 Read block 5 len 40960 25/30 Write block 283 len 4096 data: 3 26/30 Write block 175 len 12288 data: 3 3 3 27/30 Write block 584 len 32768 data: 4 4 4 3 2 3 3 3 28/30 Write block 285 len 4096 data: 4 29/30 Write block 26 len 40960 data: 5 6 7 7 6 6 6 6 5 5 30/30 Read block 197 len 24576 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 9 Done Done Done false 20 Acked 1019 Read 9 Done Done Done false 21 Acked 1020 Read 1 Done Done Done false 22 Acked 1021 Write 2 Done Done Done false 23 Acked 1022 Read 10 Done Done Done false 24 Acked 1023 Read 10 Done Done Done false 25 Acked 1024 Write 1 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Write 8 Done Done Done false 28 Acked 1027 Write 1 Done Done Done false 29 Acked 1028 Write 10 Done Done Done false 30 Acked 1029 Read 6 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 12 12 12 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:29:27.175409421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1313} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 F F F 1 020-039 30 30 30 61 61 61 T T T 2 040-059 30 30 30 61 61 61 T T T 3 060-079 23 23 23 46 46 46 F F F 4 080-099 32 32 32 66 66 66 F F F 5 100-119 27 27 27 55 55 55 F F F 6 120-139 32 32 32 65 65 65 F F F 7 140-159 32 32 32 66 66 66 F F F 8 160-179 32 32 32 66 66 66 T T T 9 180-199 32 32 32 65 65 65 F F F 10 200-219 28 28 28 59 59 59 F F F 11 220-239 25 25 25 51 51 51 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 30 30 30 61 61 61 T T T 15 300-319 26 26 26 53 53 53 F F F 16 320-339 30 30 30 62 62 62 F F F 17 340-359 32 32 32 66 66 66 F F F 18 360-379 32 32 32 66 66 66 F F F 19 380-399 31 31 31 64 64 64 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 28 28 28 57 57 57 F F F 22 440-459 28 28 28 58 58 58 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 32 32 32 66 66 66 F F F 26 520-539 25 25 25 51 51 51 F F F 27 540-559 29 29 29 60 60 60 F F F 28 560-579 32 32 32 65 65 65 F F F 29 580-599 31 31 31 64 64 64 T T T Max gen: 32, Max flush: 66 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 F F F 1 020-039 33 33 33 67 67 67 F F F 2 040-059 33 33 33 67 67 67 F F F 3 060-079 23 23 23 46 46 46 F F F 4 080-099 32 32 32 66 66 66 F F F 5 100-119 27 27 27 55 55 55 F F F 6 120-139 32 32 32 65 65 65 F F F 7 140-159 32 32 32 66 66 66 F F F 8 160-179 33 33 33 67 67 67 F F F 9 180-199 32 32 32 65 65 65 F F F 10 200-219 28 28 28 59 59 59 F F F 11 220-239 25 25 25 51 51 51 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 22 22 22 42 42 42 F F F 14 280-299 33 33 33 67 67 67 F F F 15 300-319 26 26 26 53 53 53 F F F 16 320-339 30 30 30 62 62 62 F F F 17 340-359 32 32 32 66 66 66 F F F 18 360-379 32 32 32 66 66 66 F F F 19 380-399 31 31 31 64 64 64 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 28 28 28 57 57 57 F F F 22 440-459 28 28 28 58 58 58 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 32 32 32 66 66 66 F F F 26 520-539 25 25 25 51 51 51 F F F 27 540-559 29 29 29 60 60 60 F F F 28 560-579 32 32 32 65 65 65 F F F 29 580-599 33 33 33 67 67 67 F F F Max gen: 33, Max flush: 67 Begin loop 17 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.825797567Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.826243633Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.826280359Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321} {"msg":"Crucible 63358067-4710-4a39-acb1-f6c854be8eeb has session id: f2cc1d60-364c-492e-b1d1-f21fd758b19f","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.826502468Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Upstairs opts: Upstairs UUID: 63358067-4710-4a39-acb1-f6c854be8eeb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.826538514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Crucible stats registered with UUID: 63358067-4710-4a39-acb1-f6c854be8eeb","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.826554672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.826855542Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321} {"msg":"63358067-4710-4a39-acb1-f6c854be8eeb active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.826996699Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827015627Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827030245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827044024Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"1","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827057893Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"1","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.82707905Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"2","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827105738Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"2","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827133305Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"io task","client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827160512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"io task","client":"2","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.82718658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"io task","client":"1","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:29:37.827307848Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"io task","client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:29:37.827337415Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"io task","client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:29:37.82738378Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827409768Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827434895Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"ds_connection connected from Ok(127.0.0.1:48344)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827514517Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"io task","client":"2","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"ds_connection connected from Ok(127.0.0.1:62785)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827553963Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"io task","client":"1","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.827570062Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"io task","client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.828265943Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"1","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.82829291Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"1","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.828313108Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"2","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.828337876Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"2","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.828884272Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"2","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.828913458Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"1","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.829439686Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"2","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.829469563Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"2","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.829484392Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.829496871Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"1","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.829510399Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"1","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:37.829524558Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.828266738Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"io task","client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"ds_connection connected from Ok(127.0.0.1:62141)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.828356549Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"io task","client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.829408425Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.829436702Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.829948232Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830460501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830487638Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[0]R flush_numbers[0..12]: [56, 67, 67, 46, 66, 55, 65, 66, 67, 65, 59, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830503537Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[0]R generation[0..12]: [28, 33, 33, 23, 32, 27, 32, 32, 33, 32, 28, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830518115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830532604Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[1]R flush_numbers[0..12]: [56, 67, 67, 46, 66, 55, 65, 66, 67, 65, 59, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830550462Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[1]R generation[0..12]: [28, 33, 33, 23, 32, 27, 32, 32, 33, 32, 28, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830564901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830579099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[2]R flush_numbers[0..12]: [56, 67, 67, 46, 66, 55, 65, 66, 67, 65, 59, 51]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830592968Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[2]R generation[0..12]: [28, 33, 33, 23, 32, 27, 32, 32, 33, 32, 28, 25]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830606646Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830620705Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Max found gen is 34","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:29:47.830633724Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830649602Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Next flush: 68","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830662741Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.83067579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"":"downstairs","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830689098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830701107Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"0","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830714226Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"1","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830727625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"client":"2","":"downstairs","session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830741013Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830755552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"63358067-4710-4a39-acb1-f6c854be8eeb is now active with session: f2cc1d60-364c-492e-b1d1-f21fd758b19f","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.83076858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830780779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321,"session_id":"f2cc1d60-364c-492e-b1d1-f21fd758b19f"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:29:47.830793038Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 516 len 36864 data: 4 4 3 3 3 3 2 4 4 02/30 Read block 473 len 32768 03/30 Write block 145 len 8192 data: 3 4 04/30 Write block 298 len 32768 data: 6 5 6 5 3 3 3 4 05/30 Write block 521 len 24576 data: 4 3 5 5 5 5 06/30 Read block 7 len 8192 07/30 Read block 404 len 8192 08/30 Read block 558 len 20480 09/30 Flush 10/30 Write block 229 len 4096 data: 3 11/30 Read block 374 len 4096 12/30 Flush 13/30 Write block 275 len 40960 data: 3 2 2 2 2 3 3 3 4 4 14/30 Write block 197 len 16384 data: 4 4 5 5 15/30 Flush 16/30 Write block 230 len 28672 data: 3 4 4 4 5 4 4 17/30 Write block 571 len 16384 data: 5 5 5 5 18/30 Read block 88 len 28672 19/30 Read block 480 len 32768 20/30 Read block 470 len 28672 21/30 Read block 157 len 40960 22/30 Write block 296 len 12288 data: 6 6 7 23/30 Read block 570 len 32768 24/30 Flush 25/30 Write block 428 len 16384 data: 4 4 4 4 26/30 Read block 331 len 40960 27/30 Write block 56 len 28672 data: 4 4 4 4 3 3 3 28/30 Write block 101 len 36864 data: 5 5 6 4 2 2 2 2 2 29/30 Read block 38 len 4096 30/30 Read block 279 len 36864 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 4 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Write 7 Done Done Done false 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Read 1 Done Done Done false 30 Acked 1029 Read 9 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 0 1 Done 6 5 6 17 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:29:47.93621593Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1321} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 F F F 1 020-039 33 33 33 67 67 67 F F F 2 040-059 33 33 33 67 67 67 T T T 3 060-079 23 23 23 46 46 46 T T T 4 080-099 32 32 32 66 66 66 F F F 5 100-119 27 27 27 55 55 55 T T T 6 120-139 32 32 32 65 65 65 F F F 7 140-159 34 34 34 68 68 68 F F F 8 160-179 33 33 33 67 67 67 F F F 9 180-199 34 34 34 70 70 70 F F F 10 200-219 34 34 34 70 70 70 F F F 11 220-239 34 34 34 71 71 71 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 34 34 34 70 70 70 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 34 34 34 68 68 68 F F F 16 320-339 30 30 30 62 62 62 F F F 17 340-359 32 32 32 66 66 66 F F F 18 360-379 32 32 32 66 66 66 F F F 19 380-399 31 31 31 64 64 64 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 28 28 28 57 57 57 T T T 22 440-459 28 28 28 58 58 58 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 34 34 34 68 68 68 F F F 26 520-539 34 34 34 68 68 68 F F F 27 540-559 29 29 29 60 60 60 F F F 28 560-579 34 34 34 71 71 71 F F F 29 580-599 33 33 33 67 67 67 F F F Max gen: 34, Max flush: 71 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 F F F 1 020-039 33 33 33 67 67 67 F F F 2 040-059 35 35 35 72 72 72 F F F 3 060-079 35 35 35 72 72 72 F F F 4 080-099 32 32 32 66 66 66 F F F 5 100-119 35 35 35 72 72 72 F F F 6 120-139 32 32 32 65 65 65 F F F 7 140-159 34 34 34 68 68 68 F F F 8 160-179 33 33 33 67 67 67 F F F 9 180-199 34 34 34 70 70 70 F F F 10 200-219 34 34 34 70 70 70 F F F 11 220-239 34 34 34 71 71 71 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 34 34 34 70 70 70 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 34 34 34 68 68 68 F F F 16 320-339 30 30 30 62 62 62 F F F 17 340-359 32 32 32 66 66 66 F F F 18 360-379 32 32 32 66 66 66 F F F 19 380-399 31 31 31 64 64 64 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 35 35 35 72 72 72 F F F 22 440-459 28 28 28 58 58 58 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 34 34 34 68 68 68 F F F 26 520-539 34 34 34 68 68 68 F F F 27 540-559 29 29 29 60 60 60 F F F 28 560-579 34 34 34 71 71 71 F F F 29 580-599 33 33 33 67 67 67 F F F Max gen: 35, Max flush: 72 Begin loop 18 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.440449669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.440700264Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.440725402Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329} {"msg":"Crucible 9f0a0a66-bee3-48f5-ad07-adff29513517 has session id: 1999b739-70ab-40c1-8c1f-3d51733a72d0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441193295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Upstairs opts: Upstairs UUID: 9f0a0a66-bee3-48f5-ad07-adff29513517, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441234181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Crucible stats registered with UUID: 9f0a0a66-bee3-48f5-ad07-adff29513517","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441251159Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441576767Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329} {"msg":"9f0a0a66-bee3-48f5-ad07-adff29513517 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441656559Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441687526Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"0","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441715473Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"0","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441740701Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441764688Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441789036Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"2","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441807814Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"2","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441829562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"io task","client":"2","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.44185553Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"io task","client":"0","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441880597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"io task","client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:02.441910704Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"io task","client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:02.441944561Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"io task","client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"ds_connection connected from Ok(127.0.0.1:63899)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441961799Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"io task","client":"0","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:02.441977838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.441990316Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.442008524Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"ds_connection connected from Ok(127.0.0.1:50028)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.442187757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"io task","client":"2","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.442276818Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"io task","client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.442773159Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"2","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.442797276Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"2","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.442811385Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"0","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.442825023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"0","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.443314385Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"0","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.443422564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"2","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.443750652Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"0","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.443772669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"0","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.443789838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.443919755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"2","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.443938123Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"2","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:02.443951562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.443819799Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"io task","client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"ds_connection connected from Ok(127.0.0.1:42204)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.444150756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"io task","client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.445406402Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.445447757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446034059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446612522Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446640779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[0]R flush_numbers[0..12]: [56, 67, 72, 72, 66, 72, 65, 68, 67, 70, 70, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446659997Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[0]R generation[0..12]: [28, 33, 35, 35, 32, 35, 32, 34, 33, 34, 34, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446678086Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446696134Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[1]R flush_numbers[0..12]: [56, 67, 72, 72, 66, 72, 65, 68, 67, 70, 70, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446714052Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[1]R generation[0..12]: [28, 33, 35, 35, 32, 35, 32, 34, 33, 34, 34, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.44673655Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446755348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[2]R flush_numbers[0..12]: [56, 67, 72, 72, 66, 72, 65, 68, 67, 70, 70, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446773046Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[2]R generation[0..12]: [28, 33, 35, 35, 32, 35, 32, 34, 33, 34, 34, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446790335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446813442Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446829911Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446846509Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Next flush: 73","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446862827Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446878976Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"":"downstairs","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446895734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446910483Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"0","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446927151Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"1","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446943689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"client":"2","":"downstairs","session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446960308Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446975746Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"9f0a0a66-bee3-48f5-ad07-adff29513517 is now active with session: 1999b739-70ab-40c1-8c1f-3d51733a72d0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.446993974Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.447018992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329,"session_id":"1999b739-70ab-40c1-8c1f-3d51733a72d0"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:12.44704023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 124 len 12288 data: 4 4 5 02/30 Write block 76 len 16384 data: 3 3 4 3 03/30 Write block 23 len 28672 data: 3 4 5 6 7 8 8 04/30 Read block 482 len 24576 05/30 Write block 570 len 32768 data: 3 6 6 6 6 6 7 6 06/30 Read block 481 len 16384 07/30 Read block 132 len 20480 08/30 Write block 550 len 4096 data: 5 09/30 Read block 496 len 40960 10/30 Write block 304 len 24576 data: 4 5 5 5 5 5 11/30 Read block 365 len 8192 12/30 Read block 220 len 32768 13/30 Read block 49 len 20480 14/30 Read block 480 len 20480 15/30 Write block 449 len 40960 data: 5 5 5 4 3 3 3 3 4 6 16/30 Flush 17/30 Write block 506 len 12288 data: 4 4 5 18/30 Read block 552 len 16384 19/30 Write block 122 len 40960 data: 3 4 5 5 6 4 2 2 3 3 20/30 Read block 150 len 12288 21/30 Read block 404 len 8192 22/30 Flush 23/30 Write block 568 len 40960 data: 4 3 4 7 7 7 7 7 8 7 24/30 Flush 25/30 Write block 305 len 8192 data: 6 6 26/30 Write block 568 len 36864 data: 5 4 5 8 8 8 8 8 9 27/30 Write block 189 len 4096 data: 3 28/30 Read block 8 len 24576 29/30 Read block 332 len 36864 30/30 Write block 420 len 32768 data: 3 3 3 3 4 4 4 6 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 2 Done Done Done false 26 Acked 1025 Write 9 Done Done Done false 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Read 6 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:12.544988212Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1329} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 F F F 1 020-039 36 36 36 73 73 73 F F F 2 040-059 35 35 35 72 72 72 F F F 3 060-079 36 36 36 73 73 73 F F F 4 080-099 32 32 32 66 66 66 F F F 5 100-119 35 35 35 72 72 72 F F F 6 120-139 36 36 36 74 74 74 F F F 7 140-159 34 34 34 68 68 68 F F F 8 160-179 33 33 33 67 67 67 F F F 9 180-199 34 34 34 70 70 70 T T T 10 200-219 34 34 34 70 70 70 F F F 11 220-239 34 34 34 71 71 71 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 34 34 34 70 70 70 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 36 36 36 73 73 73 T T T 16 320-339 30 30 30 62 62 62 F F F 17 340-359 32 32 32 66 66 66 F F F 18 360-379 32 32 32 66 66 66 F F F 19 380-399 31 31 31 64 64 64 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 35 35 35 72 72 72 T T T 22 440-459 36 36 36 73 73 73 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 36 36 36 74 74 74 F F F 26 520-539 34 34 34 68 68 68 F F F 27 540-559 36 36 36 73 73 73 F F F 28 560-579 36 36 36 75 75 75 T T T 29 580-599 33 33 33 67 67 67 F F F Max gen: 36, Max flush: 75 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 F F F 1 020-039 36 36 36 73 73 73 F F F 2 040-059 35 35 35 72 72 72 F F F 3 060-079 36 36 36 73 73 73 F F F 4 080-099 32 32 32 66 66 66 F F F 5 100-119 35 35 35 72 72 72 F F F 6 120-139 36 36 36 74 74 74 F F F 7 140-159 34 34 34 68 68 68 F F F 8 160-179 33 33 33 67 67 67 F F F 9 180-199 37 37 37 76 76 76 F F F 10 200-219 34 34 34 70 70 70 F F F 11 220-239 34 34 34 71 71 71 F F F 12 240-259 21 21 21 41 41 41 F F F 13 260-279 34 34 34 70 70 70 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 37 37 37 76 76 76 F F F 16 320-339 30 30 30 62 62 62 F F F 17 340-359 32 32 32 66 66 66 F F F 18 360-379 32 32 32 66 66 66 F F F 19 380-399 31 31 31 64 64 64 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 37 37 37 76 76 76 F F F 22 440-459 36 36 36 73 73 73 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 F F F 25 500-519 36 36 36 74 74 74 F F F 26 520-539 34 34 34 68 68 68 F F F 27 540-559 36 36 36 73 73 73 F F F 28 560-579 37 37 37 76 76 76 F F F 29 580-599 33 33 33 67 67 67 F F F Max gen: 37, Max flush: 76 Begin loop 19 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.089903859Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.090581012Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.090611049Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337} {"msg":"Crucible 123151a6-31b9-4943-b17b-9989a342aee8 has session id: a8ea62c2-ed72-454f-907d-becdd6922c61","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.090633657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Upstairs opts: Upstairs UUID: 123151a6-31b9-4943-b17b-9989a342aee8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.090672223Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Crucible stats registered with UUID: 123151a6-31b9-4943-b17b-9989a342aee8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.090695771Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091067724Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337} {"msg":"123151a6-31b9-4943-b17b-9989a342aee8 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091092662Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.09111249Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"0","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091138697Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"0","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091164065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091190592Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.0912154Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"2","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091240627Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"2","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091263885Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"io task","client":"0","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091287622Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"io task","client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.09131034Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"io task","client":"2","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"ds_connection connected from Ok(127.0.0.1:33657)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091333018Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"io task","client":"0","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"ds_connection connected from Ok(127.0.0.1:40935)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091357515Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"io task","client":"2","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:27.091391722Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"io task","client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:27.091421579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"io task","client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:27.091570954Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091587043Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091606711Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.091640218Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"io task","client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.092183184Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"0","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.092208781Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"0","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.09222338Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"2","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.092237578Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"2","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.092856787Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"2","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.092879724Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"0","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.093406722Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"2","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.09343097Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"2","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.093449088Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.093462097Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"0","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.093475865Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"0","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:27.093489414Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.092417797Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"io task","client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"ds_connection connected from Ok(127.0.0.1:48334)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.092845095Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"io task","client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.093756295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.093793051Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.09430358Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.094820739Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.094849756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[0]R flush_numbers[0..12]: [56, 73, 72, 73, 66, 72, 74, 68, 67, 76, 70, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.094868815Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[0]R generation[0..12]: [28, 36, 35, 36, 32, 35, 36, 34, 33, 37, 34, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.094886983Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.094905321Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[1]R flush_numbers[0..12]: [56, 73, 72, 73, 66, 72, 74, 68, 67, 76, 70, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.094923089Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[1]R generation[0..12]: [28, 36, 35, 36, 32, 35, 36, 34, 33, 37, 34, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.094944877Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.094963335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[2]R flush_numbers[0..12]: [56, 73, 72, 73, 66, 72, 74, 68, 67, 76, 70, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.094980823Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"}Wait for a query_work_queue command to finish before sending IO {"msg":"[2]R generation[0..12]: [28, 36, 35, 36, 32, 35, 36, 34, 33, 37, 34, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095003001Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095021119Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Max found gen is 38","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095037488Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095054256Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Next flush: 77","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095070484Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095086843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"":"downstairs","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095104261Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.09511641Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"0","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095129889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"1","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095143217Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"client":"2","":"downstairs","session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095156486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095168815Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"123151a6-31b9-4943-b17b-9989a342aee8 is now active with session: a8ea62c2-ed72-454f-907d-becdd6922c61","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095181753Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095195802Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337,"session_id":"a8ea62c2-ed72-454f-907d-becdd6922c61"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:37.095208371Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 248 len 4096 data: 2 02/30 Read block 137 len 36864 03/30 Read block 419 len 40960 04/30 Write block 207 len 8192 data: 9 8 05/30 Read block 89 len 24576 06/30 Write block 360 len 36864 data: 6 5 5 4 4 4 4 3 4 07/30 Read block 171 len 32768 08/30 Write block 444 len 32768 data: 7 7 7 7 6 6 6 6 09/30 Write block 131 len 8192 data: 4 5 10/30 Read block 28 len 16384 11/30 Write block 155 len 28672 data: 4 3 3 4 3 3 3 12/30 Read block 274 len 20480 13/30 Read block 598 len 4096 14/30 Write block 510 len 12288 data: 6 7 6 15/30 Flush 16/30 Write block 28 len 40960 data: 9 9 7 7 7 7 6 6 5 6 17/30 Read block 429 len 16384 18/30 Read block 471 len 40960 19/30 Read block 187 len 16384 20/30 Read block 553 len 24576 21/30 Read block 69 len 28672 22/30 Write block 100 len 28672 data: 6 6 6 7 5 3 3 23/30 Read block 114 len 8192 24/30 Read block 236 len 40960 25/30 Write block 18 len 16384 data: 5 4 3 3 26/30 Write block 497 len 16384 data: 4 3 3 3 27/30 Read block 330 len 32768 28/30 Read block 222 len 4096 29/30 Read block 234 len 8192 30/30 Read block 263 len 4096 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:15 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 16 Acked 1015 Write 10 Done Done Done false 17 Acked 1016 Read 4 Done Done Done false 18 Acked 1017 Read 10 Done Done Done false 19 Acked 1018 Read 4 Done Done Done false 20 Acked 1019 Read 6 Done Done Done false 21 Acked 1020 Read 7 Done Done Done false 22 Acked 1021 Write 7 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Read 10 Done Done Done false 25 Acked 1024 Write 4 Done Done Done false 26 Acked 1025 Write 4 Done Done Done false 27 Acked 1026 Read 8 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Read 2 Done Done Done false 30 Acked 1029 Read 1 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 15 15 15 45 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1014 1014 1014 Downstairs last five completed: 1014 1013 1012 1011 1010 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:37.144322223Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1337} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 56 56 56 T T T 1 020-039 36 36 36 73 73 73 T T T 2 040-059 35 35 35 72 72 72 F F F 3 060-079 36 36 36 73 73 73 F F F 4 080-099 32 32 32 66 66 66 F F F 5 100-119 35 35 35 72 72 72 T T T 6 120-139 38 38 38 77 77 77 F F F 7 140-159 38 38 38 77 77 77 F F F 8 160-179 38 38 38 77 77 77 F F F 9 180-199 37 37 37 76 76 76 F F F 10 200-219 38 38 38 77 77 77 F F F 11 220-239 34 34 34 71 71 71 F F F 12 240-259 38 38 38 77 77 77 F F F 13 260-279 34 34 34 70 70 70 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 37 37 37 76 76 76 F F F 16 320-339 30 30 30 62 62 62 F F F 17 340-359 32 32 32 66 66 66 F F F 18 360-379 38 38 38 77 77 77 F F F 19 380-399 31 31 31 64 64 64 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 37 37 37 76 76 76 F F F 22 440-459 38 38 38 77 77 77 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 26 26 26 53 53 53 T T T 25 500-519 38 38 38 77 77 77 T T T 26 520-539 34 34 34 68 68 68 F F F 27 540-559 36 36 36 73 73 73 F F F 28 560-579 37 37 37 76 76 76 F F F 29 580-599 33 33 33 67 67 67 F F F Max gen: 38, Max flush: 77 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 78 78 78 F F F 1 020-039 39 39 39 78 78 78 F F F 2 040-059 35 35 35 72 72 72 F F F 3 060-079 36 36 36 73 73 73 F F F 4 080-099 32 32 32 66 66 66 F F F 5 100-119 39 39 39 78 78 78 F F F 6 120-139 38 38 38 77 77 77 F F F 7 140-159 38 38 38 77 77 77 F F F 8 160-179 38 38 38 77 77 77 F F F 9 180-199 37 37 37 76 76 76 F F F 10 200-219 38 38 38 77 77 77 F F F 11 220-239 34 34 34 71 71 71 F F F 12 240-259 38 38 38 77 77 77 F F F 13 260-279 34 34 34 70 70 70 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 37 37 37 76 76 76 F F F 16 320-339 30 30 30 62 62 62 F F F 17 340-359 32 32 32 66 66 66 F F F 18 360-379 38 38 38 77 77 77 F F F 19 380-399 31 31 31 64 64 64 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 37 37 37 76 76 76 F F F 22 440-459 38 38 38 77 77 77 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 39 39 39 78 78 78 F F F 25 500-519 39 39 39 78 78 78 F F F 26 520-539 34 34 34 68 68 68 F F F 27 540-559 36 36 36 73 73 73 F F F 28 560-579 37 37 37 76 76 76 F F F 29 580-599 33 33 33 67 67 67 F F F Max gen: 39, Max flush: 78 Begin loop 20 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.707266484Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.707769914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.707800471Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345} {"msg":"Crucible 56bb8063-73de-4aef-89ef-bf7fb15a9482 has session id: 617089b6-2cf0-4b9e-8882-451ee435e7e3","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.707862975Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Upstairs opts: Upstairs UUID: 56bb8063-73de-4aef-89ef-bf7fb15a9482, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708094762Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Crucible stats registered with UUID: 56bb8063-73de-4aef-89ef-bf7fb15a9482","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708130618Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708360515Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345} {"msg":"56bb8063-73de-4aef-89ef-bf7fb15a9482 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708483303Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708502591Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"0","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.70851749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"0","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708532369Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708546147Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708559356Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"2","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708575714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"2","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708632108Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"io task","client":"0","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708653756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"io task","client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708668885Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"io task","client":"2","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:51.708799702Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"io task","client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:51.708844978Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"io task","client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:40246)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708874115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"io task","client":"0","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:39677)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.708900582Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"io task","client":"2","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:30:51.709028889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.709064176Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.709088563Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.709130989Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"io task","client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.709779035Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"2","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.709808932Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"2","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.7098272Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"0","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.709847788Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"0","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.710469366Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"2","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.710498183Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"0","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.71103325Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"0","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.711062478Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"0","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.711077866Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.711134351Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"2","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.711153689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"2","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:30:51.711167327Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.710179871Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"io task","client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"ds_connection connected from Ok(127.0.0.1:45110)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.710397569Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"io task","client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.711819099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.711849496Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.712520109Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.71311543Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713148457Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[0]R flush_numbers[0..12]: [78, 78, 72, 73, 66, 78, 77, 77, 77, 76, 77, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713168465Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[0]R generation[0..12]: [39, 39, 35, 36, 32, 39, 38, 38, 38, 37, 38, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713186623Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713204781Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[1]R flush_numbers[0..12]: [78, 78, 72, 73, 66, 78, 77, 77, 77, 76, 77, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713226669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[1]R generation[0..12]: [39, 39, 35, 36, 32, 39, 38, 38, 38, 37, 38, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713244417Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713262276Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[2]R flush_numbers[0..12]: [78, 78, 72, 73, 66, 78, 77, 77, 77, 76, 77, 71]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:31:01.713279564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[2]R generation[0..12]: [39, 39, 35, 36, 32, 39, 38, 38, 38, 37, 38, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713300112Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.71331811Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713334728Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713351376Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Next flush: 79","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713367795Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713384133Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"":"downstairs","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713400612Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.71341565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"0","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713432428Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"1","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713449217Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"client":"2","":"downstairs","session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713466475Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713483903Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"56bb8063-73de-4aef-89ef-bf7fb15a9482 is now active with session: 617089b6-2cf0-4b9e-8882-451ee435e7e3","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713500142Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.7135153Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345,"session_id":"617089b6-2cf0-4b9e-8882-451ee435e7e3"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:01.713530759Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 201 len 20480 data: 4 4 4 5 6 02/30 Read block 341 len 24576 03/30 Read block 331 len 40960 04/30 Read block 215 len 40960 05/30 Read block 432 len 12288 06/30 Read block 99 len 28672 07/30 Write block 261 len 36864 data: 2 3 3 3 3 3 3 2 3 08/30 Read block 232 len 24576 09/30 Read block 377 len 28672 10/30 Write block 392 len 16384 data: 8 8 8 7 11/30 Read block 532 len 8192 12/30 Read block 338 len 40960 13/30 Write block 422 len 24576 data: 4 4 5 5 5 7 14/30 Read block 72 len 28672 15/30 Flush 16/30 Write block 240 len 4096 data: 4 17/30 Write block 159 len 32768 data: 4 4 4 3 3 3 3 3 18/30 Write block 563 len 24576 data: 5 5 5 4 3 6 19/30 Read block 383 len 12288 20/30 Write block 595 len 20480 data: 6 5 4 4 4 21/30 Write block 335 len 12288 data: 3 3 4 22/30 Flush 23/30 Write block 90 len 24576 data: 5 5 5 6 6 6 24/30 Read block 374 len 28672 25/30 Read block 234 len 32768 26/30 Write block 211 len 32768 data: 6 5 5 4 5 4 4 3 27/30 Read block 117 len 40960 28/30 Write block 366 len 8192 data: 5 4 29/30 Read block 426 len 40960 30/30 Write block 119 len 36864 data: 3 3 3 4 5 6 6 7 5 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 6 Done Done Done false 25 Acked 1024 Read 7 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 Acked 1026 Write 8 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Write 2 Done Done Done false 30 Acked 1029 Read 10 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:31:01.742666371Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1345} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 78 78 78 F F F 1 020-039 39 39 39 78 78 78 F F F 2 040-059 35 35 35 72 72 72 F F F 3 060-079 36 36 36 73 73 73 F F F 4 080-099 32 32 32 66 66 66 T T T 5 100-119 39 39 39 78 78 78 T T T 6 120-139 38 38 38 77 77 77 T T T 7 140-159 40 40 40 80 80 80 F F F 8 160-179 40 40 40 80 80 80 F F F 9 180-199 37 37 37 76 76 76 F F F 10 200-219 40 40 40 79 79 79 T T T 11 220-239 34 34 34 71 71 71 F F F 12 240-259 40 40 40 80 80 80 F F F 13 260-279 40 40 40 79 79 79 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 37 37 37 76 76 76 F F F 16 320-339 40 40 40 81 81 81 F F F 17 340-359 32 32 32 66 66 66 F F F 18 360-379 38 38 38 77 77 77 T T T 19 380-399 40 40 40 79 79 79 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 40 40 40 79 79 79 F F F 22 440-459 38 38 38 77 77 77 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 39 39 39 78 78 78 F F F 25 500-519 39 39 39 78 78 78 F F F 26 520-539 34 34 34 68 68 68 F F F 27 540-559 36 36 36 73 73 73 F F F 28 560-579 40 40 40 81 81 81 F F F 29 580-599 40 40 40 81 81 81 F F F Max gen: 40, Max flush: 81 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 78 78 78 F F F 1 020-039 39 39 39 78 78 78 F F F 2 040-059 35 35 35 72 72 72 F F F 3 060-079 36 36 36 73 73 73 F F F 4 080-099 41 41 41 82 82 82 F F F 5 100-119 41 41 41 82 82 82 F F F 6 120-139 41 41 41 82 82 82 F F F 7 140-159 40 40 40 80 80 80 F F F 8 160-179 40 40 40 80 80 80 F F F 9 180-199 37 37 37 76 76 76 F F F 10 200-219 41 41 41 82 82 82 F F F 11 220-239 34 34 34 71 71 71 F F F 12 240-259 40 40 40 80 80 80 F F F 13 260-279 40 40 40 79 79 79 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 37 37 37 76 76 76 F F F 16 320-339 40 40 40 81 81 81 F F F 17 340-359 32 32 32 66 66 66 F F F 18 360-379 41 41 41 82 82 82 F F F 19 380-399 40 40 40 79 79 79 F F F 20 400-419 30 30 30 63 63 63 F F F 21 420-439 40 40 40 79 79 79 F F F 22 440-459 38 38 38 77 77 77 F F F 23 460-479 28 28 28 57 57 57 F F F 24 480-499 39 39 39 78 78 78 F F F 25 500-519 39 39 39 78 78 78 F F F 26 520-539 34 34 34 68 68 68 F F F 27 540-559 36 36 36 73 73 73 F F F 28 560-579 40 40 40 81 81 81 F F F 29 580-599 40 40 40 81 81 81 F F F Max gen: 41, Max flush: 82 Begin loop 21 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.976297629Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.976621887Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.976647885Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353} {"msg":"Crucible 584364bc-f1a2-4140-b3ee-9ae08ef775fd has session id: e26e3782-f7af-4241-97cb-f2e7224daf14","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977003589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Upstairs opts: Upstairs UUID: 584364bc-f1a2-4140-b3ee-9ae08ef775fd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977050175Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Crucible stats registered with UUID: 584364bc-f1a2-4140-b3ee-9ae08ef775fd","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977076602Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977392161Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353} {"msg":"584364bc-f1a2-4140-b3ee-9ae08ef775fd active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977454925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977489481Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.9775066Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977522958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"1","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977546856Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"1","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977572283Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"2","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977598751Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"2","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977622768Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"io task","client":"2","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977646826Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"io task","client":"1","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977672533Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"io task","client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:31:15.977846706Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"io task","client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"ds_connection connected from Ok(127.0.0.1:48172)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977886532Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"io task","client":"2","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:31:15.977957505Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"io task","client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"ds_connection connected from Ok(127.0.0.1:43207)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.977989232Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"io task","client":"1","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:31:15.978018739Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.978043516Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.978067894Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.978084562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"io task","client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.978583143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"2","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.97861069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"2","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.978626709Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"1","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.978640357Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"1","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.979210251Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"2","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.9793217Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"1","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.979706791Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"2","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.979737068Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"2","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.979753777Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.979766745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"1","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.979780994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"1","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:15.979794333Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.979527085Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"io task","client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"ds_connection connected from Ok(127.0.0.1:48759)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.97967361Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"io task","client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.98098861Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.981031526Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.981532846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982053565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982077032Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[0]R flush_numbers[0..12]: [78, 78, 72, 73, 82, 82, 82, 80, 80, 76, 82, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982092111Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[0]R generation[0..12]: [39, 39, 35, 36, 41, 41, 41, 40, 40, 37, 41, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982106429Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982120778Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[1]R flush_numbers[0..12]: [78, 78, 72, 73, 82, 82, 82, 80, 80, 76, 82, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982134546Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[1]R generation[0..12]: [39, 39, 35, 36, 41, 41, 41, 40, 40, 37, 41, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982152025Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982166883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[2]R flush_numbers[0..12]: [78, 78, 72, 73, 82, 82, 82, 80, 80, 76, 82, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982180672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[2]R generation[0..12]: [39, 39, 35, 36, 41, 41, 41, 40, 40, 37, 41, 34]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982194431Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982208559Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982225628Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982238876Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Next flush: 83","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982251785Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982264733Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"":"downstairs","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982277962Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982289861Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"0","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.98230307Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"1","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982316388Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"client":"2","":"downstairs","session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982330027Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982342226Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"584364bc-f1a2-4140-b3ee-9ae08ef775fd is now active with session: e26e3782-f7af-4241-97cb-f2e7224daf14","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982356924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982369083Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353,"session_id":"e26e3782-f7af-4241-97cb-f2e7224daf14"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:25.982381252Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 221 len 28672 data: 4 5 4 4 3 3 2 02/30 Flush 03/30 Write block 105 len 16384 data: 4 4 3 3 04/30 Read block 273 len 24576 05/30 Write block 478 len 4096 data: 3 06/30 Write block 110 len 28672 data: 2 2 2 2 3 3 2 07/30 Read block 486 len 4096 08/30 Flush 09/30 Write block 539 len 8192 data: 3 3 10/30 Write block 461 len 20480 data: 5 7 5 3 3 11/30 Write block 467 len 12288 data: 4 5 5 12/30 Flush 13/30 Write block 310 len 20480 data: 5 5 5 5 6 14/30 Write block 599 len 4096 data: 5 15/30 Write block 208 len 32768 data: 9 7 6 7 6 6 5 6 16/30 Read block 286 len 12288 17/30 Write block 414 len 40960 data: 2 2 2 2 2 2 4 4 5 5 18/30 Write block 12 len 40960 data: 2 3 5 5 6 6 6 5 4 4 19/30 Write block 341 len 24576 data: 5 6 6 6 5 5 20/30 Write block 320 len 8192 data: 7 7 21/30 Read block 471 len 8192 22/30 Write block 357 len 16384 data: 7 8 7 7 23/30 Flush 24/30 Write block 180 len 8192 data: 4 3 25/30 Read block 539 len 40960 26/30 Flush 27/30 Write block 500 len 12288 data: 4 3 3 28/30 Read block 436 len 8192 29/30 Write block 503 len 32768 data: 4 4 4 5 5 6 5 7 30/30 Read block 56 len 16384 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 3 Done Done Done false 29 Acked 1028 Read 2 Done Done Done false 30 Acked 1029 Write 8 Done Done Done false 31 Acked 1030 Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 1 2 Done 3 4 3 10 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:31:26.013754033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1353} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 87 87 87 F F F 1 020-039 42 42 42 87 87 87 F F F 2 040-059 35 35 35 72 72 72 F F F 3 060-079 36 36 36 73 73 73 F F F 4 080-099 41 41 41 82 82 82 F F F 5 100-119 42 42 42 84 84 84 F F F 6 120-139 41 41 41 82 82 82 F F F 7 140-159 40 40 40 80 80 80 F F F 8 160-179 40 40 40 80 80 80 F F F 9 180-199 42 42 42 88 88 88 F F F 10 200-219 42 42 42 86 86 86 F F F 11 220-239 42 42 42 83 83 83 F F F 12 240-259 40 40 40 80 80 80 F F F 13 260-279 40 40 40 79 79 79 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 42 42 42 87 87 87 F F F 17 340-359 42 42 42 87 87 87 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 40 40 40 79 79 79 F F F 20 400-419 42 42 42 87 87 87 F F F 21 420-439 42 42 42 87 87 87 F F F 22 440-459 38 38 38 77 77 77 F F F 23 460-479 42 42 42 85 85 85 F F F 24 480-499 39 39 39 78 78 78 F F F 25 500-519 39 39 39 78 78 78 T T T 26 520-539 42 42 42 85 85 85 F F F 27 540-559 42 42 42 85 85 85 F F F 28 560-579 40 40 40 81 81 81 F F F 29 580-599 42 42 42 86 86 86 F F F Max gen: 42, Max flush: 88 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 87 87 87 F F F 1 020-039 42 42 42 87 87 87 F F F 2 040-059 35 35 35 72 72 72 F F F 3 060-079 36 36 36 73 73 73 F F F 4 080-099 41 41 41 82 82 82 F F F 5 100-119 42 42 42 84 84 84 F F F 6 120-139 41 41 41 82 82 82 F F F 7 140-159 40 40 40 80 80 80 F F F 8 160-179 40 40 40 80 80 80 F F F 9 180-199 42 42 42 88 88 88 F F F 10 200-219 42 42 42 86 86 86 F F F 11 220-239 42 42 42 83 83 83 F F F 12 240-259 40 40 40 80 80 80 F F F 13 260-279 40 40 40 79 79 79 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 42 42 42 87 87 87 F F F 17 340-359 42 42 42 87 87 87 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 40 40 40 79 79 79 F F F 20 400-419 42 42 42 87 87 87 F F F 21 420-439 42 42 42 87 87 87 F F F 22 440-459 38 38 38 77 77 77 F F F 23 460-479 42 42 42 85 85 85 F F F 24 480-499 39 39 39 78 78 78 F F F 25 500-519 43 43 43 89 89 89 F F F 26 520-539 42 42 42 85 85 85 F F F 27 540-559 42 42 42 85 85 85 F F F 28 560-579 40 40 40 81 81 81 F F F 29 580-599 42 42 42 86 86 86 F F F Max gen: 43, Max flush: 89 Begin loop 22 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.188045542Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.188412245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.188440103Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361} {"msg":"Crucible 76e01aa9-a291-4f3f-88f9-5757f4a2b23d has session id: 0211d3a4-db0e-442f-88f5-c2870022bb71","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.188842203Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Upstairs opts: Upstairs UUID: 76e01aa9-a291-4f3f-88f9-5757f4a2b23d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.188960711Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Crucible stats registered with UUID: 76e01aa9-a291-4f3f-88f9-5757f4a2b23d","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.188991968Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189258881Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361} {"msg":"76e01aa9-a291-4f3f-88f9-5757f4a2b23d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189386209Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189403857Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"0","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189418586Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"0","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189432474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189454102Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.18947952Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"2","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189506487Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"2","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189530464Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"io task","client":"0","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189555832Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"io task","client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189580099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"io task","client":"2","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:31:40.189723955Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"io task","client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:31:40.189763701Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"io task","client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"ds_connection connected from Ok(127.0.0.1:62453)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189796398Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"io task","client":"2","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:31:40.189826695Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189842224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.189863922Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"ds_connection connected from Ok(127.0.0.1:34625)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.18987993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"io task","client":"0","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.190053553Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"io task","client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.190562562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"2","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.190597269Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"2","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.190664782Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"0","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.190739015Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"0","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.191077471Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"2","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.191175881Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"0","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.191558783Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"2","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.19159587Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"2","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.191628596Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.191655794Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"0","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.191682801Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"0","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:40.191703339Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.190929811Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"io task","client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"ds_connection connected from Ok(127.0.0.1:33254)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.191114422Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"io task","client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.192017083Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.192056069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.192579547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193111624Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193138232Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[0]R flush_numbers[0..12]: [87, 87, 72, 73, 82, 84, 82, 80, 80, 88, 86, 83]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.19315731Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[0]R generation[0..12]: [42, 42, 35, 36, 41, 42, 41, 40, 40, 42, 42, 42]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193175378Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193193556Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[1]R flush_numbers[0..12]: [87, 87, 72, 73, 82, 84, 82, 80, 80, 88, 86, 83]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193221474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[1]R generation[0..12]: [42, 42, 35, 36, 41, 42, 41, 40, 40, 42, 42, 42]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193243411Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.19326153Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[2]R flush_numbers[0..12]: [87, 87, 72, 73, 82, 84, 82, 80, 80, 88, 86, 83]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193278928Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {Wait for a query_work_queue command to finish before sending IO "msg":"[2]R generation[0..12]: [42, 42, 35, 36, 41, 42, 41, 40, 40, 42, 42, 42]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193298976Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193316924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Max found gen is 44","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193333242Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193349851Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Next flush: 90","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193366099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193382368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"":"downstairs","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193398946Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193413974Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"0","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193430713Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"1","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193447211Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"client":"2","":"downstairs","session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.19346364Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193478928Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"76e01aa9-a291-4f3f-88f9-5757f4a2b23d is now active with session: 0211d3a4-db0e-442f-88f5-c2870022bb71","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193495246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:31:50.193511855Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361,"session_id":"0211d3a4-db0e-442f-88f5-c2870022bb71"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2024-07-09T21:31:50.193527233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 175 len 16384 data: 4 4 4 3 02/30 Write block 222 len 20480 data: 6 5 5 4 4 03/30 Flush 04/30 Write block 192 len 32768 data: 7 7 7 6 4 5 5 6 05/30 Read block 433 len 36864 06/30 Read block 6 len 16384 07/30 Write block 143 len 4096 data: 3 08/30 Write block 112 len 16384 data: 3 3 4 4 09/30 Flush 10/30 Write block 419 len 24576 data: 3 5 5 6 6 6 11/30 Flush 12/30 Write block 97 len 12288 data: 8 7 7 13/30 Read block 46 len 4096 14/30 Write block 59 len 36864 data: 5 4 4 4 2 2 2 2 3 15/30 Read block 21 len 16384 16/30 Read block 464 len 4096 17/30 Flush 18/30 Write block 415 len 28672 data: 3 3 3 3 4 6 6 19/30 Write block 508 len 4096 data: 7 20/30 Write block 181 len 12288 data: 4 3 3 21/30 Write block 558 len 36864 data: 5 4 4 6 7 6 6 6 5 22/30 Read block 198 len 20480 23/30 Flush 24/30 Write block 345 len 8192 data: 6 6 25/30 Read block 589 len 12288 26/30 Write block 262 len 20480 data: 4 4 4 4 4 27/30 Read block 293 len 20480 28/30 Write block 589 len 36864 data: 4 4 4 3 4 6 7 6 5 29/30 Read block 195 len 8192 30/30 Read block 449 len 28672 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 2 Done Done Done false 25 Acked 1024 Read 3 Done Done Done false 26 Acked 1025 Write 5 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Read 2 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:31:50.225218462Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1361} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 87 87 87 F F F 1 020-039 42 42 42 87 87 87 F F F 2 040-059 44 44 44 93 93 93 F F F 3 060-079 44 44 44 93 93 93 F F F 4 080-099 44 44 44 93 93 93 F F F 5 100-119 44 44 44 91 91 91 F F F 6 120-139 41 41 41 82 82 82 F F F 7 140-159 44 44 44 91 91 91 F F F 8 160-179 44 44 44 90 90 90 F F F 9 180-199 44 44 44 94 94 94 F F F 10 200-219 42 42 42 86 86 86 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 40 40 40 80 80 80 F F F 13 260-279 40 40 40 79 79 79 T T T 14 280-299 34 34 34 71 71 71 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 42 42 42 87 87 87 F F F 17 340-359 42 42 42 87 87 87 T T T 18 360-379 42 42 42 87 87 87 F F F 19 380-399 40 40 40 79 79 79 F F F 20 400-419 44 44 44 94 94 94 F F F 21 420-439 44 44 44 94 94 94 F F F 22 440-459 38 38 38 77 77 77 F F F 23 460-479 42 42 42 85 85 85 F F F 24 480-499 39 39 39 78 78 78 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 42 42 42 85 85 85 F F F 27 540-559 44 44 44 94 94 94 F F F 28 560-579 44 44 44 94 94 94 F F F 29 580-599 42 42 42 86 86 86 T T T Max gen: 44, Max flush: 94 A Difference in extent metadata is expected here On loop 22 Verifying data now No change after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 87 87 87 F F F 1 020-039 42 42 42 87 87 87 F F F 2 040-059 44 44 44 93 93 93 F F F 3 060-079 44 44 44 93 93 93 F F F 4 080-099 44 44 44 93 93 93 F F F 5 100-119 44 44 44 91 91 91 F F F 6 120-139 41 41 41 82 82 82 F F F 7 140-159 44 44 44 91 91 91 F F F 8 160-179 44 44 44 90 90 90 F F F 9 180-199 44 44 44 94 94 94 F F F 10 200-219 42 42 42 86 86 86 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 40 40 40 80 80 80 F F F 13 260-279 45 45 45 95 95 95 F F F 14 280-299 34 34 34 71 71 71 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 42 42 42 87 87 87 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 40 40 40 79 79 79 F F F 20 400-419 44 44 44 94 94 94 F F F 21 420-439 44 44 44 94 94 94 F F F 22 440-459 38 38 38 77 77 77 F F F 23 460-479 42 42 42 85 85 85 F F F 24 480-499 39 39 39 78 78 78 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 42 42 42 85 85 85 F F F 27 540-559 44 44 44 94 94 94 F F F 28 560-579 44 44 44 94 94 94 F F F 29 580-599 45 45 45 95 95 95 F F F Max gen: 45, Max flush: 95 Begin loop 23 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.426935783Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.427231834Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.427250672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369} {"msg":"Crucible 1847d9f0-a455-41a8-9299-f3a066265ae9 has session id: c2e01d35-ae0f-405c-a099-98fe0b54e373","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.427653282Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Upstairs opts: Upstairs UUID: 1847d9f0-a455-41a8-9299-f3a066265ae9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.427699148Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Crucible stats registered with UUID: 1847d9f0-a455-41a8-9299-f3a066265ae9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.427727495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428002918Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369} {"msg":"1847d9f0-a455-41a8-9299-f3a066265ae9 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428114797Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428140994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"0","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428158212Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"0","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428172841Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.42818669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428200668Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"2","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428215077Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"2","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428236785Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"io task","client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428262962Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"io task","client":"2","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428289779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"io task","client":"0","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"ds_connection connected from Ok(127.0.0.1:36665)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428460952Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"io task","client":"0","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:04.428601908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"io task","client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:04.428647024Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"io task","client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:04.428668812Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"ds_connection connected from Ok(127.0.0.1:64630)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.42868235Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"io task","client":"2","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428697349Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428710567Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.428809178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"io task","client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.429322677Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"2","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.429360153Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"2","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.42939044Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"0","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.429453014Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"0","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.429921027Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"2","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.430055624Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"0","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.430350675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"2","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.430373003Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"2","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.430387461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.430560604Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"0","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.430579242Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"0","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:04.430592771Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.429577756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"io task","client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"ds_connection connected from Ok(127.0.0.1:49421)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.429788116Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"io task","client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.431031143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.43105635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.431546721Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432069909Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432091498Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[0]R flush_numbers[0..12]: [87, 87, 93, 93, 93, 91, 82, 91, 90, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432106246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[0]R generation[0..12]: [42, 42, 44, 44, 44, 44, 41, 44, 44, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432120635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432134953Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[1]R flush_numbers[0..12]: [87, 87, 93, 93, 93, 91, 82, 91, 90, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432148822Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[1]R generation[0..12]: [42, 42, 44, 44, 44, 44, 41, 44, 44, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.43216269Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432181749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[2]R flush_numbers[0..12]: [87, 87, 93, 93, 93, 91, 82, 91, 90, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432196087Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[2]R generation[0..12]: [42, 42, 44, 44, 44, 44, 41, 44, 44, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432209876Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432223934Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432240213Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432253671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Next flush: 96","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.43226664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432279589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"":"downstairs","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432293007Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432304996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"0","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432318385Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"1","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432331804Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"client":"2","":"downstairs","session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432345412Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432357871Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"1847d9f0-a455-41a8-9299-f3a066265ae9 is now active with session: c2e01d35-ae0f-405c-a099-98fe0b54e373","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.43237074Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432384738Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369,"session_id":"c2e01d35-ae0f-405c-a099-98fe0b54e373"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:14.432397207Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 565 len 40960 data: 7 6 4 7 5 6 9 9 9 9 02/30 Read block 487 len 16384 03/30 Read block 269 len 32768 04/30 Write block 68 len 16384 data: 3 4 4 5 05/30 Write block 428 len 40960 data: 5 5 5 5 4 5 5 4 3 3 06/30 Read block 32 len 28672 07/30 Write block 487 len 20480 data: 4 5 5 6 5 08/30 Write block 587 len 32768 data: 4 3 5 5 5 4 5 7 09/30 Read block 311 len 32768 10/30 Write block 111 len 28672 data: 3 4 4 5 5 3 3 11/30 Read block 350 len 24576 12/30 Read block 221 len 12288 13/30 Write block 535 len 8192 data: 3 3 14/30 Read block 325 len 36864 15/30 Write block 151 len 24576 data: 3 3 3 3 5 4 16/30 Read block 546 len 36864 17/30 Write block 58 len 36864 data: 5 6 5 5 5 3 3 3 3 18/30 Flush 19/30 Flush 20/30 Write block 451 len 28672 data: 7 5 4 4 4 4 5 21/30 Write block 169 len 12288 data: 2 2 3 22/30 Read block 231 len 36864 23/30 Read block 43 len 36864 24/30 Write block 416 len 4096 data: 4 25/30 Write block 288 len 20480 data: 5 5 3 3 3 26/30 Write block 255 len 8192 data: 4 4 27/30 Write block 486 len 24576 data: 4 5 6 6 7 6 28/30 Write block 487 len 4096 data: 6 29/30 Write block 463 len 16384 data: 6 4 4 4 30/30 Write block 579 len 32768 data: 3 3 4 5 4 5 5 5 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 7 Done Done Done false 21 Acked 1020 Write 3 Done Done Done false 22 Acked 1021 Read 9 Done Done Done false 23 Acked 1022 Read 9 Done Done Done false 24 Acked 1023 Write 1 Done Done Done false 25 Acked 1024 Write 5 Done Done Done false 26 Acked 1025 Write 2 Done Done Done false 27 Acked 1026 Write 6 Done Done Done false 28 Acked 1027 Write 1 Sent Done Sent false 29 Acked 1028 Write 4 Sent Done Sent false 30 Acked 1029 Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 3 1 3 7 Done 8 10 8 26 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:14.458373643Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1369} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 87 87 87 F F F 1 020-039 42 42 42 87 87 87 F F F 2 040-059 46 46 46 96 96 96 F F F 3 060-079 46 46 46 96 96 96 F F F 4 080-099 44 44 44 93 93 93 F F F 5 100-119 46 46 46 96 96 96 F F F 6 120-139 41 41 41 82 82 82 F F F 7 140-159 46 46 46 96 96 96 F F F 8 160-179 44 44 44 90 90 90 T T T 9 180-199 44 44 44 94 94 94 F F F 10 200-219 42 42 42 86 86 86 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 40 40 40 80 80 80 T T T 13 260-279 45 45 45 95 95 95 F F F 14 280-299 34 34 34 71 71 71 T T T 15 300-319 42 42 42 86 86 86 F F F 16 320-339 42 42 42 87 87 87 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 40 40 40 79 79 79 F F F 20 400-419 44 44 44 94 94 94 T T T 21 420-439 46 46 46 96 96 96 F F F 22 440-459 38 38 38 77 77 77 T T T 23 460-479 42 42 42 85 85 85 T T T 24 480-499 46 46 46 96 96 96 T T T 25 500-519 44 44 44 94 94 94 F F F 26 520-539 46 46 46 96 96 96 F F F 27 540-559 44 44 44 94 94 94 F F F 28 560-579 46 46 46 96 96 96 T T T 29 580-599 46 46 46 96 96 96 T T T Max gen: 46, Max flush: 96 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 87 87 87 F F F 1 020-039 42 42 42 87 87 87 F F F 2 040-059 46 46 46 96 96 96 F F F 3 060-079 46 46 46 96 96 96 F F F 4 080-099 44 44 44 93 93 93 F F F 5 100-119 46 46 46 96 96 96 F F F 6 120-139 41 41 41 82 82 82 F F F 7 140-159 46 46 46 96 96 96 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 44 44 44 94 94 94 F F F 10 200-219 42 42 42 86 86 86 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 47 47 47 97 97 97 F F F 13 260-279 45 45 45 95 95 95 F F F 14 280-299 47 47 47 97 97 97 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 42 42 42 87 87 87 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 40 40 40 79 79 79 F F F 20 400-419 47 47 47 97 97 97 F F F 21 420-439 46 46 46 96 96 96 F F F 22 440-459 47 47 47 97 97 97 F F F 23 460-479 47 47 47 97 97 97 F F F 24 480-499 47 47 47 97 97 97 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 46 46 46 96 96 96 F F F 27 540-559 44 44 44 94 94 94 F F F 28 560-579 47 47 47 97 97 97 F F F 29 580-599 47 47 47 97 97 97 F F F Max gen: 47, Max flush: 97 Begin loop 24 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.743147632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.743425514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.743448732Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377} {"msg":"Crucible 60820fd1-7dbb-4698-bc05-b962c6d4d9d1 has session id: 734eb438-f4c4-45f5-8f39-aaf3a0a93f1a","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744011486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Upstairs opts: Upstairs UUID: 60820fd1-7dbb-4698-bc05-b962c6d4d9d1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744046893Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Crucible stats registered with UUID: 60820fd1-7dbb-4698-bc05-b962c6d4d9d1","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744062341Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744360731Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377} {"msg":"60820fd1-7dbb-4698-bc05-b962c6d4d9d1 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744404907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744427655Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"0","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744450663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"0","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.74447346Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"1","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744496838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"1","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744519656Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744544923Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744570381Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"io task","client":"0","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744626005Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"io task","client":"1","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744655452Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"io task","client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"ds_connection connected from Ok(127.0.0.1:53048)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744701688Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"io task","client":"1","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"ds_connection connected from Ok(127.0.0.1:57849)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.744732785Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"io task","client":"0","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:28.744997299Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"io task","client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:28.745027196Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"io task","client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:28.745044474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.745057073Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.745070491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.74518597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"io task","client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.745490239Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"1","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.745524076Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"1","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.745548684Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"0","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.745570302Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"0","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.746115358Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"1","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.746154714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"0","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.746713418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"1","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.746743035Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"1","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.746770473Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.746785271Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"0","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.74679901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"0","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:28.746812528Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.74625037Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"io task","client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"ds_connection connected from Ok(127.0.0.1:62491)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.74635431Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"io task","client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.747652581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.747680418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748193097Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748719845Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748747952Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[0]R flush_numbers[0..12]: [87, 87, 96, 96, 93, 96, 82, 96, 97, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748767021Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[0]R generation[0..12]: [42, 42, 46, 46, 44, 46, 41, 46, 47, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748785229Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748803397Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[1]R flush_numbers[0..12]: [87, 87, 96, 96, 93, 96, 82, 96, 97, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748820675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[1]R generation[0..12]: [42, 42, 46, 46, 44, 46, 41, 46, 47, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748842713Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748860681Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[2]R flush_numbers[0..12]: [87, 87, 96, 96, 93, 96, 82, 96, 97, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.74887787Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[2]R generation[0..12]: [42, 42, 46, 46, 44, 46, 41, 46, 47, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748898668Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748916436Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748932854Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748949543Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Next flush: 98","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748965881Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748981979Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"":"downstairs","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.748998578Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.749013586Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"0","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.749030514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"1","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.749047093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"client":"2","":"downstairs","session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.749063711Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.74907909Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"60820fd1-7dbb-4698-bc05-b962c6d4d9d1 is now active with session: 734eb438-f4c4-45f5-8f39-aaf3a0a93f1a","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.749097478Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.749112606Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377,"session_id":"734eb438-f4c4-45f5-8f39-aaf3a0a93f1a"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:38.749128025Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 259 len 8192 data: 4 2 02/30 Write block 440 len 24576 data: 6 7 7 7 8 8 03/30 Read block 289 len 20480 04/30 Read block 482 len 16384 05/30 Read block 259 len 4096 06/30 Write block 287 len 36864 data: 4 6 6 4 4 4 3 4 5 07/30 Read block 481 len 28672 08/30 Read block 518 len 20480 09/30 Write block 12 len 24576 data: 3 4 6 6 7 7 10/30 Read block 83 len 40960 11/30 Read block 428 len 24576 12/30 Write block 147 len 36864 data: 3 3 3 3 4 4 4 4 6 13/30 Read block 89 len 40960 14/30 Write block 425 len 24576 data: 6 6 8 6 6 6 15/30 Read block 142 len 36864 16/30 Read block 466 len 24576 17/30 Read block 319 len 40960 18/30 Write block 584 len 24576 data: 6 6 6 5 4 6 19/30 Read block 53 len 40960 20/30 Read block 258 len 24576 21/30 Write block 557 len 32768 data: 5 6 5 5 7 8 7 7 22/30 Flush 23/30 Write block 389 len 32768 data: 8 7 9 9 9 9 8 5 24/30 Write block 467 len 4096 data: 5 25/30 Write block 535 len 24576 data: 4 4 3 3 4 4 26/30 Write block 66 len 28672 data: 4 4 4 5 5 6 5 27/30 Write block 411 len 4096 data: 4 28/30 Write block 540 len 32768 data: 5 3 3 3 3 3 5 5 29/30 Read block 184 len 36864 30/30 Write block 566 len 12288 data: 7 5 8 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 8 Done Done Done false 24 Acked 1023 Write 1 Done Done Done false 25 Acked 1024 Write 6 Done Done Done false 26 Acked 1025 Write 7 Done Done Done false 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Write 8 Sent Sent Done false 29 Acked 1028 Read 9 Sent Sent Done false 30 Acked 1029 Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 3 3 1 7 Done 5 5 7 17 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:38.775673663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1377} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 42 42 42 87 87 87 F F F 2 040-059 46 46 46 96 96 96 F F F 3 060-079 46 46 46 96 96 96 T T T 4 080-099 44 44 44 93 93 93 F F F 5 100-119 46 46 46 96 96 96 F F F 6 120-139 41 41 41 82 82 82 F F F 7 140-159 48 48 48 98 98 98 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 44 44 44 94 94 94 F F F 10 200-219 42 42 42 86 86 86 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 48 48 48 98 98 98 F F F 13 260-279 48 48 48 98 98 98 F F F 14 280-299 48 48 48 98 98 98 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 42 42 42 87 87 87 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 40 40 40 79 79 79 T T T 20 400-419 47 47 47 97 97 97 T T T 21 420-439 48 48 48 98 98 98 F F F 22 440-459 48 48 48 98 98 98 F F F 23 460-479 47 47 47 97 97 97 T T T 24 480-499 47 47 47 97 97 97 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 46 46 46 96 96 96 T T T 27 540-559 48 48 48 98 98 98 T T T 28 560-579 48 48 48 98 98 98 T T T 29 580-599 48 48 48 98 98 98 F F F Max gen: 48, Max flush: 98 A Difference in extent metadata is expected here On loop 24 Verifying data now No change after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 42 42 42 87 87 87 F F F 2 040-059 46 46 46 96 96 96 F F F 3 060-079 49 49 49 99 99 99 F F F 4 080-099 44 44 44 93 93 93 F F F 5 100-119 46 46 46 96 96 96 F F F 6 120-139 41 41 41 82 82 82 F F F 7 140-159 48 48 48 98 98 98 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 44 44 44 94 94 94 F F F 10 200-219 42 42 42 86 86 86 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 48 48 48 98 98 98 F F F 13 260-279 48 48 48 98 98 98 F F F 14 280-299 48 48 48 98 98 98 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 42 42 42 87 87 87 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 49 49 49 99 99 99 F F F 20 400-419 49 49 49 99 99 99 F F F 21 420-439 48 48 48 98 98 98 F F F 22 440-459 48 48 48 98 98 98 F F F 23 460-479 49 49 49 99 99 99 F F F 24 480-499 47 47 47 97 97 97 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 49 49 49 99 99 99 F F F 27 540-559 49 49 49 99 99 99 F F F 28 560-579 49 49 49 99 99 99 F F F 29 580-599 48 48 48 98 98 98 F F F Max gen: 49, Max flush: 99 Begin loop 25 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.037035383Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.037382768Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.037401936Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385} {"msg":"Crucible b7f8eb36-0cf7-4a37-8724-feb9dfe56922 has session id: b8e3c611-e50e-4bde-b271-cb9955a3a2c7","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.037678869Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Upstairs opts: Upstairs UUID: b7f8eb36-0cf7-4a37-8724-feb9dfe56922, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.037758911Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Crucible stats registered with UUID: b7f8eb36-0cf7-4a37-8724-feb9dfe56922","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.037785328Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038190268Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385} {"msg":"b7f8eb36-0cf7-4a37-8724-feb9dfe56922 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038215246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038230684Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038245153Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038259011Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"1","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.03827301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"1","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038286919Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"2","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038300727Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"2","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038314206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"io task","client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038337234Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"io task","client":"1","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038364391Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"io task","client":"2","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:53.038516126Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"io task","client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:53.038559262Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"io task","client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:32:53.038590829Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038611516Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038636324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"ds_connection connected from Ok(127.0.0.1:47484)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038701087Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"io task","client":"1","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"ds_connection connected from Ok(127.0.0.1:61869)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038753232Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"io task","client":"2","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.038887059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"io task","client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.03937931Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"1","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.039411047Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"1","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.039437414Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"2","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.039462652Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"2","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.040017797Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"2","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.040143315Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"1","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.040515188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"2","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.040537956Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"2","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.040555474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.040630836Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"1","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.040649714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"1","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:32:53.040663383Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.040236721Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"io task","client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"ds_connection connected from Ok(127.0.0.1:38193)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.040565489Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"io task","client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.041605375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.041642372Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042217655Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042736663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042761961Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[0]R flush_numbers[0..12]: [98, 87, 96, 99, 93, 96, 82, 98, 97, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042780849Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[0]R generation[0..12]: [48, 42, 46, 49, 44, 46, 41, 48, 47, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042798737Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042816986Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[1]R flush_numbers[0..12]: [98, 87, 96, 99, 93, 96, 82, 98, 97, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042834334Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[1]R generation[0..12]: [48, 42, 46, 49, 44, 46, 41, 48, 47, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042855982Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.04287518Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[2]R flush_numbers[0..12]: [98, 87, 96, 99, 93, 96, 82, 98, 97, 94, 86, 90]","v":0,"name":"crucible","level":30,"timeWait for a query_work_queue command to finish before sending IO ":"2024-07-09T21:33:03.042893348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[2]R generation[0..12]: [48, 42, 46, 49, 44, 46, 41, 48, 47, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042913746Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042931774Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042948432Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042965311Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Next flush: 100","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042981739Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.042997917Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"":"downstairs","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.043014636Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.043030094Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"0","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.043046753Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"1","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.043063281Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"client":"2","":"downstairs","session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.043080129Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.043095458Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"b7f8eb36-0cf7-4a37-8724-feb9dfe56922 is now active with session: b8e3c611-e50e-4bde-b271-cb9955a3a2c7","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.043114086Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.043130084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385,"session_id":"b8e3c611-e50e-4bde-b271-cb9955a3a2c7"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:03.043144003Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 339 len 4096 data: 4 02/30 Write block 56 len 32768 data: 5 5 6 7 6 6 6 4 03/30 Write block 36 len 8192 data: 6 7 04/30 Read block 463 len 28672 05/30 Read block 305 len 32768 06/30 Read block 501 len 24576 07/30 Read block 367 len 12288 08/30 Write block 526 len 36864 data: 6 5 5 4 3 2 2 3 3 09/30 Read block 457 len 8192 10/30 Write block 548 len 40960 data: 4 4 6 6 5 4 4 3 5 6 11/30 Write block 399 len 4096 data: 3 12/30 Write block 248 len 12288 data: 3 2 2 13/30 Write block 142 len 12288 data: 3 4 3 14/30 Read block 61 len 20480 15/30 Read block 443 len 20480 16/30 Write block 29 len 40960 data: 10 8 8 8 8 7 7 7 8 5 17/30 Write block 385 len 36864 data: 4 4 4 7 9 8 10 10 10 18/30 Read block 529 len 8192 19/30 Read block 54 len 40960 20/30 Write block 462 len 40960 data: 8 7 5 5 5 6 6 6 5 4 21/30 Write block 408 len 40960 data: 7 8 8 5 5 3 3 4 5 4 22/30 Write block 53 len 36864 data: 3 4 4 6 6 7 8 7 7 23/30 Write block 132 len 40960 data: 6 5 6 6 4 3 4 3 3 3 24/30 Write block 485 len 28672 data: 3 5 7 7 7 8 7 25/30 Read block 175 len 20480 26/30 Read block 296 len 12288 27/30 Read block 411 len 4096 28/30 Read block 458 len 28672 29/30 Read block 157 len 28672 30/30 Read block 468 len 28672 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 10 Done Done Done false 23 Acked 1022 Write 9 Done Done Done false 24 Acked 1023 Write 10 Done Done Done false 25 Acked 1024 Write 7 Done Done Done false 26 Acked 1025 Read 5 Done Done Done false 27 Acked 1026 Read 3 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Read 7 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Read 7 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 0 1 Done 10 9 10 29 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:33:03.068063553Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1385} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 50 50 50 100 100 100 F F F 2 040-059 50 50 50 100 100 100 T T T 3 060-079 50 50 50 100 100 100 T T T 4 080-099 44 44 44 93 93 93 F F F 5 100-119 46 46 46 96 96 96 F F F 6 120-139 41 41 41 82 82 82 T T T 7 140-159 50 50 50 100 100 100 T T T 8 160-179 47 47 47 97 97 97 F F F 9 180-199 44 44 44 94 94 94 F F F 10 200-219 42 42 42 86 86 86 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 50 50 50 100 100 100 F F F 13 260-279 48 48 48 98 98 98 F F F 14 280-299 48 48 48 98 98 98 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 50 50 50 100 100 100 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 50 50 50 100 100 100 F F F 20 400-419 49 49 49 99 99 99 T T T 21 420-439 48 48 48 98 98 98 F F F 22 440-459 48 48 48 98 98 98 F F F 23 460-479 50 50 50 100 100 100 F F F 24 480-499 47 47 47 97 97 97 T T T 25 500-519 44 44 44 94 94 94 F F F 26 520-539 50 50 50 100 100 100 F F F 27 540-559 50 50 50 100 100 100 F F F 28 560-579 49 49 49 99 99 99 F F F 29 580-599 48 48 48 98 98 98 F F F Max gen: 50, Max flush: 100 A Difference in extent metadata is expected here On loop 25 Verifying data now No change after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 50 50 50 100 100 100 F F F 2 040-059 51 51 51 101 101 101 F F F 3 060-079 51 51 51 101 101 101 F F F 4 080-099 44 44 44 93 93 93 F F F 5 100-119 46 46 46 96 96 96 F F F 6 120-139 51 51 51 101 101 101 F F F 7 140-159 51 51 51 101 101 101 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 44 44 44 94 94 94 F F F 10 200-219 42 42 42 86 86 86 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 50 50 50 100 100 100 F F F 13 260-279 48 48 48 98 98 98 F F F 14 280-299 48 48 48 98 98 98 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 50 50 50 100 100 100 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 50 50 50 100 100 100 F F F 20 400-419 51 51 51 101 101 101 F F F 21 420-439 48 48 48 98 98 98 F F F 22 440-459 48 48 48 98 98 98 F F F 23 460-479 50 50 50 100 100 100 F F F 24 480-499 51 51 51 101 101 101 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 50 50 50 100 100 100 F F F 27 540-559 50 50 50 100 100 100 F F F 28 560-579 49 49 49 99 99 99 F F F 29 580-599 48 48 48 98 98 98 F F F Max gen: 51, Max flush: 101 Begin loop 26 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.30410793Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.304361385Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.304383233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393} {"msg":"Crucible 5d1c82d4-d324-4ab6-bada-e904e582e51e has session id: ce135ad2-6fbb-4ae2-948b-04727a1013b2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.304862115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Upstairs opts: Upstairs UUID: 5d1c82d4-d324-4ab6-bada-e904e582e51e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.304905811Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Crucible stats registered with UUID: 5d1c82d4-d324-4ab6-bada-e904e582e51e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.304930019Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305240778Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393} {"msg":"5d1c82d4-d324-4ab6-bada-e904e582e51e active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305267165Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305281514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305296352Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305310211Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"1","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.30532386Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"1","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305338148Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"2","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305353837Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"2","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305437018Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"io task","client":"1","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305457796Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"io task","client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305471995Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"io task","client":"2","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:40193)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305640698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"io task","client":"1","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:61703)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.305792553Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"io task","client":"2","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:33:17.305832049Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"io task","client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:33:17.305861956Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"io task","client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:33:17.306009792Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.306035169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.306050868Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.306169236Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"io task","client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.30653048Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"2","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.306556447Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"2","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.306571836Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"1","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.306585875Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"1","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.307193674Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"1","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.307218182Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"2","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.307784756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"2","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.307825982Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"2","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.307856089Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.307875647Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"1","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.307889875Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"1","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:17.307903384Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.307850005Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"io task","client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:50950)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.308049815Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"io task","client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.30890445Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.308932998Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.309508021Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.3101251Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310151617Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[0]R flush_numbers[0..12]: [98, 100, 101, 101, 93, 96, 101, 101, 97, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310170555Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[0]R generation[0..12]: [48, 50, 51, 51, 44, 46, 51, 51, 47, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310188453Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310206281Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[1]R flush_numbers[0..12]: [98, 100, 101, 101, 93, 96, 101, 101, 97, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310228539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[1]R generation[0..12]: [48, 50, 51, 51, 44, 46, 51, 51, 47, 44, 42, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310246057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310263546Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[2]R flush_numbers[0..12]: [98, 100, 101, 101, 93, 96, 101, 101, 97, 94, 86, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310290243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[2]R generation[0..12]: [48, 50, 51, 51, 44, 46, 51, 51, 47, 44, 42, 44]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:33:27.31032063Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310352967Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310374475Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310397222Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Next flush: 102","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.31042569Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310456707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"":"downstairs","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310478065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310497403Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"0","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.31052629Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"1","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310558457Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"client":"2","":"downstairs","session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310577675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310607312Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"5d1c82d4-d324-4ab6-bada-e904e582e51e is now active with session: ce135ad2-6fbb-4ae2-948b-04727a1013b2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310638209Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310667606Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393,"session_id":"ce135ad2-6fbb-4ae2-948b-04727a1013b2"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:27.310685564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 37 len 28672 data: 9 6 6 5 3 3 5 02/30 Read block 510 len 32768 03/30 Write block 139 len 8192 data: 4 4 04/30 Read block 273 len 36864 05/30 Write block 535 len 4096 data: 5 06/30 Write block 275 len 8192 data: 4 3 07/30 Read block 507 len 16384 08/30 Read block 542 len 36864 09/30 Read block 273 len 12288 10/30 Write block 408 len 8192 data: 8 9 11/30 Write block 72 len 8192 data: 6 5 12/30 Write block 59 len 20480 data: 9 8 8 7 5 13/30 Read block 561 len 12288 14/30 Read block 278 len 32768 15/30 Write block 273 len 20480 data: 4 3 5 4 3 16/30 Read block 78 len 16384 17/30 Write block 198 len 32768 data: 6 7 6 5 5 5 6 7 18/30 Write block 104 len 24576 data: 6 5 5 4 4 3 19/30 Read block 334 len 24576 20/30 Write block 548 len 8192 data: 5 5 21/30 Flush 22/30 Write block 190 len 20480 data: 5 7 8 8 8 23/30 Write block 421 len 40960 data: 7 7 7 7 7 7 9 7 7 7 24/30 Read block 239 len 16384 25/30 Write block 197 len 24576 data: 6 7 8 7 6 6 26/30 Read block 508 len 40960 27/30 Write block 35 len 12288 data: 8 8 10 28/30 Read block 482 len 12288 29/30 Read block 306 len 8192 30/30 Write block 123 len 24576 data: 6 7 7 8 6 3 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 5 Done Done Done false 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Read 4 Done Done Done false 25 Acked 1024 Write 6 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Write 3 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Read 2 Done Done Done false 30 Acked 1029 Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:33:27.335912224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1393} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 52 52 52 102 102 102 T T T 2 040-059 52 52 52 102 102 102 F F F 3 060-079 52 52 52 102 102 102 F F F 4 080-099 44 44 44 93 93 93 F F F 5 100-119 52 52 52 102 102 102 F F F 6 120-139 52 52 52 102 102 102 T T T 7 140-159 52 52 52 102 102 102 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 52 52 52 102 102 102 T T T 10 200-219 52 52 52 102 102 102 T T T 11 220-239 44 44 44 90 90 90 F F F 12 240-259 50 50 50 100 100 100 F F F 13 260-279 52 52 52 102 102 102 F F F 14 280-299 48 48 48 98 98 98 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 50 50 50 100 100 100 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 50 50 50 100 100 100 F F F 20 400-419 52 52 52 102 102 102 F F F 21 420-439 48 48 48 98 98 98 T T T 22 440-459 48 48 48 98 98 98 F F F 23 460-479 50 50 50 100 100 100 F F F 24 480-499 51 51 51 101 101 101 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 52 52 52 102 102 102 F F F 27 540-559 52 52 52 102 102 102 F F F 28 560-579 49 49 49 99 99 99 F F F 29 580-599 48 48 48 98 98 98 F F F Max gen: 52, Max flush: 102 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 52 52 52 102 102 102 F F F 3 060-079 52 52 52 102 102 102 F F F 4 080-099 44 44 44 93 93 93 F F F 5 100-119 52 52 52 102 102 102 F F F 6 120-139 53 53 53 103 103 103 F F F 7 140-159 52 52 52 102 102 102 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 53 53 53 103 103 103 F F F 10 200-219 53 53 53 103 103 103 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 50 50 50 100 100 100 F F F 13 260-279 52 52 52 102 102 102 F F F 14 280-299 48 48 48 98 98 98 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 50 50 50 100 100 100 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 50 50 50 100 100 100 F F F 20 400-419 52 52 52 102 102 102 F F F 21 420-439 53 53 53 103 103 103 F F F 22 440-459 48 48 48 98 98 98 F F F 23 460-479 50 50 50 100 100 100 F F F 24 480-499 51 51 51 101 101 101 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 52 52 52 102 102 102 F F F 27 540-559 52 52 52 102 102 102 F F F 28 560-579 49 49 49 99 99 99 F F F 29 580-599 48 48 48 98 98 98 F F F Max gen: 53, Max flush: 103 Begin loop 27 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.564932759Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.565383965Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.565415981Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401} {"msg":"Crucible 4f86e30f-2a31-44e0-a28b-aa6acd947042 has session id: 35e92e28-d062-49d3-a7eb-8329e42d6aea","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.565617212Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Upstairs opts: Upstairs UUID: 4f86e30f-2a31-44e0-a28b-aa6acd947042, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.565666637Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Crucible stats registered with UUID: 4f86e30f-2a31-44e0-a28b-aa6acd947042","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.565694504Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.56603124Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401} {"msg":"4f86e30f-2a31-44e0-a28b-aa6acd947042 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566055288Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566077566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"0","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566104663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"0","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566130081Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"1","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566155178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"1","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566177746Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566225471Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566296734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"io task","client":"0","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566331411Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"io task","client":"1","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566357188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"io task","client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"ds_connection connected from Ok(127.0.0.1:48559)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566385545Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"io task","client":"1","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"ds_connection connected from Ok(127.0.0.1:59251)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566633281Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"io task","client":"0","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:33:41.566678236Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"io task","client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:33:41.566701594Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"io task","client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:33:41.566718592Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566731421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.56674492Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.566851069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"io task","client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.567093755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"1","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.567117983Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"1","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.567217673Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"0","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.56724689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"0","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.567558569Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"1","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.567666418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"0","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.568101905Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"1","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.568128503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"1","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.568143761Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.568163849Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"0","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.568178178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"0","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:41.568192116Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.567602389Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"io task","client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"ds_connection connected from Ok(127.0.0.1:39548)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.567778542Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"io task","client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.568944156Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.568983382Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.569553686Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570161505Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570182393Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[0]R flush_numbers[0..12]: [98, 103, 102, 102, 93, 102, 103, 102, 97, 103, 103, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570197722Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[0]R generation[0..12]: [48, 53, 52, 52, 44, 52, 53, 52, 47, 53, 53, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.57021218Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570227069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[1]R flush_numbers[0..12]: [98, 103, 102, 102, 93, 102, 103, 102, 97, 103, 103, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570245407Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[1]R generation[0..12]: [48, 53, 52, 52, 44, 52, 53, 52, 47, 53, 53, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570259746Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570274064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[2]R flush_numbers[0..12]: [98, 103, 102, 102, 93, 102, 103, 102, 97, 103, 103, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570288103Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[2]R generation[0..12]: [48, 53, 52, 52, 44, 52, 53, 52, 47, 53, 53, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570301831Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.57031625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":Wait for a query_work_queue command to finish before sending IO 30,"time":"2024-07-09T21:33:51.570332328Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570346157Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Next flush: 104","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570359056Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570371964Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"":"downstairs","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570385243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570397152Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"0","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.57042028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"1","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570445787Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"client":"2","":"downstairs","session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570467355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570488883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"4f86e30f-2a31-44e0-a28b-aa6acd947042 is now active with session: 35e92e28-d062-49d3-a7eb-8329e42d6aea","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570512461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570535448Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401,"session_id":"35e92e28-d062-49d3-a7eb-8329e42d6aea"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:33:51.570557546Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 535 len 20480 data: 6 5 4 4 5 02/30 Read block 266 len 8192 03/30 Read block 213 len 32768 04/30 Read block 72 len 28672 05/30 Read block 539 len 8192 06/30 Read block 279 len 40960 07/30 Read block 136 len 40960 08/30 Write block 298 len 8192 data: 8 6 09/30 Read block 444 len 16384 10/30 Read block 307 len 36864 11/30 Read block 514 len 36864 12/30 Write block 411 len 40960 data: 6 6 4 4 5 6 5 4 5 7 13/30 Read block 224 len 24576 14/30 Read block 269 len 28672 15/30 Read block 122 len 8192 16/30 Read block 454 len 32768 17/30 Write block 538 len 4096 data: 5 18/30 Flush 19/30 Flush 20/30 Write block 462 len 20480 data: 9 8 6 6 6 21/30 Read block 59 len 32768 22/30 Write block 86 len 12288 data: 6 5 6 23/30 Read block 393 len 36864 24/30 Write block 195 len 16384 data: 7 5 7 8 25/30 Read block 356 len 32768 26/30 Read block 481 len 24576 27/30 Read block 399 len 12288 28/30 Write block 111 len 36864 data: 4 5 5 6 6 4 4 3 4 29/30 Read block 354 len 12288 30/30 Read block 264 len 12288 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 5 Done Done Done false 21 Acked 1020 Read 8 Done Done Done false 22 Acked 1021 Write 3 Done Done Done false 23 Acked 1022 Read 9 Done Done Done false 24 Acked 1023 Write 4 Done Done Done false 25 Acked 1024 Read 8 Done Done Done false 26 Acked 1025 Read 6 Done Done Done false 27 Acked 1026 Read 3 Done Done Done false 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Read 3 Done Done Done false 30 Acked 1029 Read 3 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 1 2 Done 10 11 10 31 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:33:51.594757408Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1401} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 52 52 52 102 102 102 F F F 3 060-079 52 52 52 102 102 102 F F F 4 080-099 44 44 44 93 93 93 T T T 5 100-119 52 52 52 102 102 102 T T T 6 120-139 53 53 53 103 103 103 F F F 7 140-159 52 52 52 102 102 102 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 53 53 53 103 103 103 T T T 10 200-219 53 53 53 103 103 103 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 50 50 50 100 100 100 F F F 13 260-279 52 52 52 102 102 102 F F F 14 280-299 54 54 54 104 104 104 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 50 50 50 100 100 100 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 50 50 50 100 100 100 F F F 20 400-419 54 54 54 104 104 104 F F F 21 420-439 54 54 54 104 104 104 F F F 22 440-459 48 48 48 98 98 98 F F F 23 460-479 50 50 50 100 100 100 T T T 24 480-499 51 51 51 101 101 101 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 54 54 54 104 104 104 F F F 27 540-559 52 52 52 102 102 102 F F F 28 560-579 49 49 49 99 99 99 F F F 29 580-599 48 48 48 98 98 98 F F F Max gen: 54, Max flush: 104 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 52 52 52 102 102 102 F F F 3 060-079 52 52 52 102 102 102 F F F 4 080-099 55 55 55 105 105 105 F F F 5 100-119 55 55 55 105 105 105 F F F 6 120-139 53 53 53 103 103 103 F F F 7 140-159 52 52 52 102 102 102 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 55 55 55 105 105 105 F F F 10 200-219 53 53 53 103 103 103 F F F 11 220-239 44 44 44 90 90 90 F F F 12 240-259 50 50 50 100 100 100 F F F 13 260-279 52 52 52 102 102 102 F F F 14 280-299 54 54 54 104 104 104 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 50 50 50 100 100 100 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 50 50 50 100 100 100 F F F 20 400-419 54 54 54 104 104 104 F F F 21 420-439 54 54 54 104 104 104 F F F 22 440-459 48 48 48 98 98 98 F F F 23 460-479 55 55 55 105 105 105 F F F 24 480-499 51 51 51 101 101 101 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 54 54 54 104 104 104 F F F 27 540-559 52 52 52 102 102 102 F F F 28 560-579 49 49 49 99 99 99 F F F 29 580-599 48 48 48 98 98 98 F F F Max gen: 55, Max flush: 105 Begin loop 28 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.812084441Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.812711649Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.812744635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409} {"msg":"Crucible bb0506eb-0538-4c90-ac78-ffcfe3d12b26 has session id: b6654743-72e8-41b7-bbef-781b221310b9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.812845815Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Upstairs opts: Upstairs UUID: bb0506eb-0538-4c90-ac78-ffcfe3d12b26, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.812990061Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Crucible stats registered with UUID: bb0506eb-0538-4c90-ac78-ffcfe3d12b26","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813037146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813296231Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409} {"msg":"bb0506eb-0538-4c90-ac78-ffcfe3d12b26 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813394031Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813419039Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"0","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813435717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"0","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813449856Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"1","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813463714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"1","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813477093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813490741Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.81350396Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"io task","client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813518009Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"io task","client":"1","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813531857Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"io task","client":"0","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:50518)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813713489Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"io task","client":"0","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:36512)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.813761774Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"io task","client":"1","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:05.813991042Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"io task","client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:05.814034858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"io task","client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:05.814060035Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.814083863Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.81411082Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.814129878Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"io task","client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.81451162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"1","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.814542787Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"1","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.814560055Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"0","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.814583693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"0","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.815148527Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"0","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.815260336Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"1","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.815606102Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"0","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.815635999Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"0","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.815651147Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.815663756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"1","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.815677845Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"1","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:05.815691433Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.815544723Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"io task","client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:34176)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.816040784Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"io task","client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.816909368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.816946874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.817489591Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818058744Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818080792Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[0]R flush_numbers[0..12]: [98, 103, 102, 102, 105, 105, 103, 102, 97, 105, 103, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818096051Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[0]R generation[0..12]: [48, 53, 52, 52, 55, 55, 53, 52, 47, 55, 53, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818110039Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818124428Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[1]R flush_numbers[0..12]: [98, 103, 102, 102, 105, 105, 103, 102, 97, 105, 103, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818142186Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[1]R generation[0..12]: [48, 53, 52, 52, 55, 55, 53, 52, 47, 55, 53, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818156334Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818171773Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[2]R flush_numbers[0..12]: [98, 103, 102, 102, 105, 105, 103, 102, 97, 105, 103, 90]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818185912Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[2]R generation[0..12]: [48, 53, 52, 52, 55, 55, 53, 52, 47, 55, 53, 44]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.81819951Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818223518Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:34:15.818246346Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818269413Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Next flush: 106","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818287592Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.81830364Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"":"downstairs","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818327188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818349705Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"0","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818375473Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"1","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818391071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"client":"2","":"downstairs","session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.81840784Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818432947Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"bb0506eb-0538-4c90-ac78-ffcfe3d12b26 is now active with session: b6654743-72e8-41b7-bbef-781b221310b9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818459064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818478043Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409,"session_id":"b6654743-72e8-41b7-bbef-781b221310b9"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:15.818490821Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 123 len 16384 data: 7 8 8 9 03/30 Read block 524 len 20480 04/30 Read block 170 len 32768 05/30 Write block 283 len 12288 data: 5 5 5 06/30 Flush 07/30 Write block 454 len 32768 data: 5 5 5 6 7 6 6 6 08/30 Flush 09/30 Write block 422 len 36864 data: 8 8 8 8 8 10 8 8 8 10/30 Write block 87 len 24576 data: 6 7 6 6 6 6 11/30 Read block 537 len 8192 12/30 Write block 490 len 8192 data: 9 8 13/30 Write block 189 len 28672 data: 4 6 8 9 9 9 8 14/30 Flush 15/30 Write block 83 len 4096 data: 5 16/30 Write block 216 len 20480 data: 5 5 4 4 3 17/30 Read block 188 len 32768 18/30 Read block 111 len 32768 19/30 Read block 569 len 4096 20/30 Read block 301 len 40960 21/30 Read block 496 len 32768 22/30 Write block 568 len 28672 data: 9 6 7 10 10 10 10 23/30 Read block 59 len 24576 24/30 Read block 278 len 12288 25/30 Read block 15 len 40960 26/30 Read block 436 len 36864 27/30 Write block 253 len 4096 data: 3 28/30 Read block 576 len 40960 29/30 Write block 133 len 20480 data: 6 7 7 5 4 30/30 Read block 408 len 12288 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:16 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 15 Acked 1014 Write 1 Done Done Done false 16 Acked 1015 Write 5 Done Done Done false 17 Acked 1016 Read 8 Done Done Done false 18 Acked 1017 Read 8 Done Done Done false 19 Acked 1018 Read 1 Done Done Done false 20 Acked 1019 Read 10 Done Done Done false 21 Acked 1020 Read 8 Done Done Done false 22 Acked 1021 Write 7 Done Done Done false 23 Acked 1022 Read 6 Done Done Done false 24 Acked 1023 Read 3 Done Done Done false 25 Acked 1024 Read 10 Done Done Done false 26 Acked 1025 Read 9 Done Done Done false 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Write 5 Done Done Done false 30 Acked 1029 Read 3 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 16 16 16 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1013 1013 1013 Downstairs last five completed: 1013 1012 1011 1010 1009 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:15.846644241Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1409} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 52 52 52 102 102 102 F F F 3 060-079 52 52 52 102 102 102 F F F 4 080-099 56 56 56 109 109 109 T T T 5 100-119 55 55 55 105 105 105 F F F 6 120-139 56 56 56 107 107 107 T T T 7 140-159 52 52 52 102 102 102 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 56 56 56 109 109 109 F F F 10 200-219 53 53 53 103 103 103 T T T 11 220-239 44 44 44 90 90 90 T T T 12 240-259 50 50 50 100 100 100 T T T 13 260-279 52 52 52 102 102 102 F F F 14 280-299 56 56 56 107 107 107 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 50 50 50 100 100 100 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 50 50 50 100 100 100 F F F 20 400-419 54 54 54 104 104 104 F F F 21 420-439 56 56 56 109 109 109 F F F 22 440-459 56 56 56 108 108 108 F F F 23 460-479 56 56 56 108 108 108 F F F 24 480-499 56 56 56 109 109 109 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 54 54 54 104 104 104 F F F 27 540-559 52 52 52 102 102 102 F F F 28 560-579 49 49 49 99 99 99 T T T 29 580-599 48 48 48 98 98 98 F F F Max gen: 56, Max flush: 109 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 52 52 52 102 102 102 F F F 3 060-079 52 52 52 102 102 102 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 55 55 55 105 105 105 F F F 6 120-139 57 57 57 110 110 110 F F F 7 140-159 52 52 52 102 102 102 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 56 56 56 109 109 109 F F F 10 200-219 57 57 57 110 110 110 F F F 11 220-239 57 57 57 110 110 110 F F F 12 240-259 57 57 57 110 110 110 F F F 13 260-279 52 52 52 102 102 102 F F F 14 280-299 56 56 56 107 107 107 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 50 50 50 100 100 100 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 42 42 42 87 87 87 F F F 19 380-399 50 50 50 100 100 100 F F F 20 400-419 54 54 54 104 104 104 F F F 21 420-439 56 56 56 109 109 109 F F F 22 440-459 56 56 56 108 108 108 F F F 23 460-479 56 56 56 108 108 108 F F F 24 480-499 56 56 56 109 109 109 F F F 25 500-519 44 44 44 94 94 94 F F F 26 520-539 54 54 54 104 104 104 F F F 27 540-559 52 52 52 102 102 102 F F F 28 560-579 57 57 57 110 110 110 F F F 29 580-599 48 48 48 98 98 98 F F F Max gen: 57, Max flush: 110 Begin loop 29 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.087874106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.08823527Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.088257798Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417} {"msg":"Crucible bd3275f7-c486-43b8-9579-a2560839581d has session id: 753fff79-921d-401d-a77f-d84066917907","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.088531841Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Upstairs opts: Upstairs UUID: bd3275f7-c486-43b8-9579-a2560839581d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.088573186Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Crucible stats registered with UUID: bd3275f7-c486-43b8-9579-a2560839581d","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.088597314Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.088905913Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417} {"msg":"bd3275f7-c486-43b8-9579-a2560839581d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.088957208Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.088974457Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.088989275Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089002864Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"1","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089022582Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"1","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089046899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"2","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089085296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"2","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089111033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"io task","client":"2","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089136441Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"io task","client":"1","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089165648Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"io task","client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"ds_connection connected from Ok(127.0.0.1:52128)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089276167Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"io task","client":"2","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:30.089317583Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"io task","client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:30.08934556Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"io task","client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:30.089374037Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089397435Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089420722Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"ds_connection connected from Ok(127.0.0.1:64048)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.08944603Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"io task","client":"1","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.089554619Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"io task","client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.090104025Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"2","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.090130192Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"2","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.090144861Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"1","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.090158929Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"1","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.090799396Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"2","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.090823243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"1","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.091401776Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"1","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.091428294Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"1","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.091442942Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.091455441Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"2","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.09146899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"2","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:30.091482488Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.091327438Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"io task","client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"ds_connection connected from Ok(127.0.0.1:60733)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.091819549Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"io task","client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.092761925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.092790333Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.093311801Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.093874915Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.093898443Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[0]R flush_numbers[0..12]: [98, 103, 102, 102, 110, 105, 110, 102, 97, 109, 110, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.093916501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[0]R generation[0..12]: [48, 53, 52, 52, 57, 55, 57, 52, 47, 56, 57, 57]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.093934089Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.093957137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[1]R flush_numbers[0..12]: [98, 103, 102, 102, 110, 105, 110, 102, 97, 109, 110, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.093975375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[1]R generation[0..12]: [48, 53, 52, 52, 57, 55, 57, 52, 47, 56, 57, 57]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.093992883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094010752Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[2]R flush_numbers[0..12]: [98, 103, 102, 102, 110, 105, 110, 102, 97, 109, 110, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.09402826Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[2]R generation[0..12]: [48, 53, 52, 52, 57, 55, 57, 52, 47, 56, 57, 57]","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2024-07-09T21:34:40.094048368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094066476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094082745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094099413Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Next flush: 111","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094115701Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.09413214Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"":"downstairs","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094149028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094164246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"0","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094180715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"1","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094199463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"client":"2","":"downstairs","session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094216891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.09423235Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"bd3275f7-c486-43b8-9579-a2560839581d is now active with session: 753fff79-921d-401d-a77f-d84066917907","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094248728Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094263676Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417,"session_id":"753fff79-921d-401d-a77f-d84066917907"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:40.094279335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 498 len 40960 data: 4 4 5 4 4 5 5 5 6 6 02/30 Write block 187 len 4096 data: 3 03/30 Flush 04/30 Write block 334 len 8192 data: 4 4 05/30 Read block 450 len 4096 06/30 Flush 07/30 Write block 385 len 28672 data: 5 5 5 8 10 9 11 08/30 Write block 523 len 8192 data: 6 6 09/30 Write block 383 len 40960 data: 4 3 6 6 6 9 11 10 12 11 10/30 Read block 97 len 8192 11/30 Write block 53 len 20480 data: 4 5 5 7 7 12/30 Read block 522 len 24576 13/30 Read block 11 len 40960 14/30 Read block 39 len 32768 15/30 Write block 365 len 36864 data: 5 6 5 5 4 4 2 2 3 16/30 Flush 17/30 Write block 269 len 12288 data: 4 3 3 18/30 Read block 229 len 36864 19/30 Read block 35 len 4096 20/30 Write block 590 len 28672 data: 6 6 5 6 8 8 7 21/30 Read block 180 len 4096 22/30 Flush 23/30 Flush 24/30 Write block 543 len 40960 data: 4 4 4 6 6 6 6 7 7 6 25/30 Read block 435 len 12288 26/30 Read block 496 len 28672 27/30 Write block 131 len 12288 data: 5 7 7 28/30 Write block 227 len 40960 data: 3 2 4 4 5 5 5 6 5 5 29/30 Read block 585 len 36864 30/30 Read block 478 len 8192 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 10 Done Done Done false 25 Acked 1024 Read 3 Done Done Done false 26 Acked 1025 Read 7 Done Done Done false 27 Acked 1026 Write 3 Done Done Done false 28 Acked 1027 Write 10 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Read 2 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:40.123375341Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1417} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 58 58 58 113 113 113 F F F 3 060-079 52 52 52 102 102 102 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 55 55 55 105 105 105 F F F 6 120-139 57 57 57 110 110 110 T T T 7 140-159 52 52 52 102 102 102 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 58 58 58 111 111 111 F F F 10 200-219 57 57 57 110 110 110 F F F 11 220-239 57 57 57 110 110 110 T T T 12 240-259 57 57 57 110 110 110 F F F 13 260-279 58 58 58 114 114 114 F F F 14 280-299 56 56 56 107 107 107 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 58 58 58 112 112 112 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 58 58 58 113 113 113 F F F 19 380-399 58 58 58 113 113 113 F F F 20 400-419 54 54 54 104 104 104 F F F 21 420-439 56 56 56 109 109 109 F F F 22 440-459 56 56 56 108 108 108 F F F 23 460-479 56 56 56 108 108 108 F F F 24 480-499 58 58 58 111 111 111 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 58 58 58 113 113 113 F F F 27 540-559 52 52 52 102 102 102 T T T 28 560-579 57 57 57 110 110 110 F F F 29 580-599 58 58 58 114 114 114 F F F Max gen: 58, Max flush: 114 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 58 58 58 113 113 113 F F F 3 060-079 52 52 52 102 102 102 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 55 55 55 105 105 105 F F F 6 120-139 59 59 59 115 115 115 F F F 7 140-159 52 52 52 102 102 102 F F F 8 160-179 47 47 47 97 97 97 F F F 9 180-199 58 58 58 111 111 111 F F F 10 200-219 57 57 57 110 110 110 F F F 11 220-239 59 59 59 115 115 115 F F F 12 240-259 57 57 57 110 110 110 F F F 13 260-279 58 58 58 114 114 114 F F F 14 280-299 56 56 56 107 107 107 F F F 15 300-319 42 42 42 86 86 86 F F F 16 320-339 58 58 58 112 112 112 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 58 58 58 113 113 113 F F F 19 380-399 58 58 58 113 113 113 F F F 20 400-419 54 54 54 104 104 104 F F F 21 420-439 56 56 56 109 109 109 F F F 22 440-459 56 56 56 108 108 108 F F F 23 460-479 56 56 56 108 108 108 F F F 24 480-499 58 58 58 111 111 111 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 58 58 58 113 113 113 F F F 27 540-559 59 59 59 115 115 115 F F F 28 560-579 57 57 57 110 110 110 F F F 29 580-599 58 58 58 114 114 114 F F F Max gen: 59, Max flush: 115 Begin loop 30 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.328267148Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.328499215Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.328527512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425} {"msg":"Crucible 46c3e322-d647-4e3b-a2bc-b5ed71a0a00d has session id: 88afbf0d-8397-4d37-aea7-222a85f7f2bc","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.328917634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Upstairs opts: Upstairs UUID: 46c3e322-d647-4e3b-a2bc-b5ed71a0a00d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.328962959Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Crucible stats registered with UUID: 46c3e322-d647-4e3b-a2bc-b5ed71a0a00d","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.328983707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329407905Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425} {"msg":"46c3e322-d647-4e3b-a2bc-b5ed71a0a00d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329430193Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329444122Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"0","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.32945858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"0","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329473249Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329486888Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329500316Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"2","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329517614Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"2","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329541812Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"io task","client":"0","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329567889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"io task","client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329591987Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"io task","client":"2","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:39107)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.329869199Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"io task","client":"0","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:54.330004516Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"io task","client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:54.330041673Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"io task","client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:34:54.33007095Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.330091868Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.33016459Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:49617)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.330189268Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"io task","client":"2","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.330205786Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"io task","client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.330497687Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"2","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.330532424Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"2","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.330560401Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"0","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.330586639Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"0","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.331055872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"0","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.33117285Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"2","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.33157777Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"0","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.331601528Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"0","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.331615877Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.331640714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"2","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.331654973Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"2","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:34:54.331668481Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.331156847Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"io task","client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:53656)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.331356547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"io task","client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.332403594Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.332435971Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.332977387Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333499935Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333523333Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[0]R flush_numbers[0..12]: [98, 103, 113, 102, 110, 105, 115, 102, 97, 111, 110, 115]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333538531Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[0]R generation[0..12]: [48, 53, 58, 52, 57, 55, 59, 52, 47, 58, 57, 59]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.33355316Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333571968Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[1]R flush_numbers[0..12]: [98, 103, 113, 102, 110, 105, 115, 102, 97, 111, 110, 115]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333586906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[1]R generation[0..12]: [48, 53, 58, 52, 57, 55, 59, 52, 47, 58, 57, 59]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333600855Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333615034Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[2]R flush_numbers[0..12]: [98, 103, 113, 102, 110, 105, 115, 102, 97, 111, 110, 115]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333628892Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[2]R generation[0..12]: [48, 53, 58, 52, 57, 55, 59, 52, 47, 58, 57, 59]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333642541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333656619Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Max found gen is 60","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2024-07-09T21:35:04.333669888Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Generation requested: 60 >= found:60","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333686956Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Next flush: 116","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333700285Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333713224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"":"downstairs","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333726283Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333738241Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"0","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.33375155Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"1","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333764779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"client":"2","":"downstairs","session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333779937Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333792436Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"46c3e322-d647-4e3b-a2bc-b5ed71a0a00d is now active with session: 88afbf0d-8397-4d37-aea7-222a85f7f2bc","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333805495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333817614Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425,"session_id":"88afbf0d-8397-4d37-aea7-222a85f7f2bc"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:04.333829872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 558 len 32768 data: 7 6 6 8 9 8 8 8 02/30 Read block 407 len 24576 03/30 Read block 52 len 16384 04/30 Read block 435 len 16384 05/30 Read block 320 len 16384 06/30 Write block 374 len 28672 data: 4 4 3 3 3 4 4 07/30 Write block 391 len 28672 data: 13 12 11 10 9 6 4 08/30 Write block 226 len 40960 data: 5 4 3 5 5 6 6 6 7 6 09/30 Write block 160 len 4096 data: 5 10/30 Read block 438 len 8192 11/30 Read block 22 len 8192 12/30 Write block 203 len 24576 data: 6 7 8 8 10 10 13/30 Write block 209 len 36864 data: 8 7 8 7 7 6 7 6 6 14/30 Write block 369 len 12288 data: 5 5 3 15/30 Read block 255 len 12288 16/30 Read block 293 len 20480 17/30 Write block 274 len 32768 data: 4 6 5 4 3 3 4 4 18/30 Write block 551 len 24576 data: 8 7 5 5 4 6 19/30 Read block 97 len 36864 20/30 Read block 337 len 24576 21/30 Read block 385 len 24576 22/30 Write block 300 len 28672 data: 7 6 4 4 5 7 7 23/30 Flush 24/30 Flush 25/30 Write block 451 len 28672 data: 8 6 5 6 6 6 7 26/30 Write block 482 len 32768 data: 2 3 3 4 6 8 8 8 27/30 Read block 466 len 8192 28/30 Write block 114 len 16384 data: 7 7 5 5 29/30 Read block 178 len 20480 30/30 Read block 510 len 8192 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 7 Done Done Done false 26 Acked 1025 Write 8 Done Done Done false 27 Acked 1026 Read 2 Done Done Done false 28 Acked 1027 Write 4 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Read 2 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 0 1 Done 6 5 6 17 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:04.363056606Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1425} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 58 58 58 113 113 113 F F F 3 060-079 52 52 52 102 102 102 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 55 55 55 105 105 105 T T T 6 120-139 59 59 59 115 115 115 F F F 7 140-159 52 52 52 102 102 102 F F F 8 160-179 60 60 60 116 116 116 F F F 9 180-199 58 58 58 111 111 111 F F F 10 200-219 60 60 60 116 116 116 F F F 11 220-239 60 60 60 116 116 116 F F F 12 240-259 57 57 57 110 110 110 F F F 13 260-279 60 60 60 116 116 116 F F F 14 280-299 60 60 60 116 116 116 F F F 15 300-319 60 60 60 116 116 116 F F F 16 320-339 58 58 58 112 112 112 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 60 60 60 116 116 116 F F F 19 380-399 60 60 60 116 116 116 F F F 20 400-419 54 54 54 104 104 104 F F F 21 420-439 56 56 56 109 109 109 F F F 22 440-459 56 56 56 108 108 108 T T T 23 460-479 56 56 56 108 108 108 F F F 24 480-499 58 58 58 111 111 111 T T T 25 500-519 58 58 58 111 111 111 F F F 26 520-539 58 58 58 113 113 113 F F F 27 540-559 60 60 60 116 116 116 F F F 28 560-579 60 60 60 116 116 116 F F F 29 580-599 58 58 58 114 114 114 F F F Max gen: 60, Max flush: 116 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 98 98 98 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 58 58 58 113 113 113 F F F 3 060-079 52 52 52 102 102 102 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 61 61 61 117 117 117 F F F 6 120-139 59 59 59 115 115 115 F F F 7 140-159 52 52 52 102 102 102 F F F 8 160-179 60 60 60 116 116 116 F F F 9 180-199 58 58 58 111 111 111 F F F 10 200-219 60 60 60 116 116 116 F F F 11 220-239 60 60 60 116 116 116 F F F 12 240-259 57 57 57 110 110 110 F F F 13 260-279 60 60 60 116 116 116 F F F 14 280-299 60 60 60 116 116 116 F F F 15 300-319 60 60 60 116 116 116 F F F 16 320-339 58 58 58 112 112 112 F F F 17 340-359 45 45 45 95 95 95 F F F 18 360-379 60 60 60 116 116 116 F F F 19 380-399 60 60 60 116 116 116 F F F 20 400-419 54 54 54 104 104 104 F F F 21 420-439 56 56 56 109 109 109 F F F 22 440-459 61 61 61 117 117 117 F F F 23 460-479 56 56 56 108 108 108 F F F 24 480-499 61 61 61 117 117 117 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 58 58 58 113 113 113 F F F 27 540-559 60 60 60 116 116 116 F F F 28 560-579 60 60 60 116 116 116 F F F 29 580-599 58 58 58 114 114 114 F F F Max gen: 61, Max flush: 117 Begin loop 31 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.564893765Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.565207994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.565238821Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433} {"msg":"Crucible d9b4f567-d661-49cb-8b6a-81030da65210 has session id: 1de913b8-76e5-4705-9484-69ab1605b5e8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.565566229Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Upstairs opts: Upstairs UUID: d9b4f567-d661-49cb-8b6a-81030da65210, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.565612124Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Crucible stats registered with UUID: d9b4f567-d661-49cb-8b6a-81030da65210","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.565639641Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.565915014Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433} {"msg":"d9b4f567-d661-49cb-8b6a-81030da65210 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.565981727Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566008245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"0","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566025603Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"0","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566041341Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"1","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.56605588Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"1","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566069899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566083427Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566096806Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"io task","client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566110685Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"io task","client":"0","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566124413Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"io task","client":"1","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:18.566292597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"io task","client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:18.566333463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"io task","client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"ds_connection connected from Ok(127.0.0.1:38493)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.56635961Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"io task","client":"1","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:18.566387237Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566407745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566429913Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.566527053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"io task","client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"ds_connection connected from Ok(127.0.0.1:59033)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.56656513Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"io task","client":"0","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.567037383Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"1","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.56706468Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"1","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.567196897Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"0","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.5672707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"0","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.567644813Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"1","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.56777479Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"0","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.568136704Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"1","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.568163621Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"1","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.56818114Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.56827501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"0","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.568296828Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"0","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:18.568313636Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.567140938Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"io task","client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"ds_connection connected from Ok(127.0.0.1:62877)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.567290013Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"io task","client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.568688955Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.568731171Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.569321552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.569865818Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.569891496Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[0]R flush_numbers[0..12]: [98, 103, 113, 102, 110, 117, 115, 102, 116, 111, 116, 116]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.569910864Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[0]R generation[0..12]: [48, 53, 58, 52, 57, 61, 59, 52, 60, 58, 60, 60]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.569929092Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.5699486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[1]R flush_numbers[0..12]: [98, 103, 113, 102, 110, 117, 115, 102, 116, 111, 116, 116]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.569971788Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[1]R generation[0..12]: [48, 53, 58, 52, 57, 61, 59, 52, 60, 58, 60, 60]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.569989806Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570007674Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[2]R flush_numbers[0..12]: [98, 103, 113, 102, 110, 117, 115, 102, 116, 111, 116, 116]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570025322Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[2]R generation[0..12]: [48, 53, 58, 52, 57, 61, 59, 52, 60, 58, 60, 60]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570042521Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570063069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570079507Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570096166Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Next flush: 118","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570112634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570129512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"":"downstairs","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570143711Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.57015568Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"0","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570169268Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"1","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570182777Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"client":"2","":"downstairs","session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570198295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570210964Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"d9b4f567-d661-49cb-8b6a-81030da65210 is now active with session: 1de913b8-76e5-4705-9484-69ab1605b5e8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570223993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.570236002Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:28.57024842Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 379 len 36864 data: 5 5 4 3 5 4 7 7 7 02/30 Write block 142 len 12288 data: 4 5 4 03/30 Read block 308 len 36864 04/30 Read block 486 len 32768 05/30 Read block 65 len 40960 06/30 Write block 347 len 12288 data: 7 5 4 07/30 Write block 273 len 40960 data: 5 5 7 6 5 4 4 5 5 4 08/30 Read block 376 len 36864 09/30 Write block 432 len 28672 data: 5 6 6 5 4 4 5 10/30 Read block 455 len 24576 11/30 Write block 190 len 4096 data: 7 12/30 Read block 508 len 8192 13/30 Write block 71 len 16384 data: 7 7 6 5 14/30 Read block 131 len 36864 15/30 Write block 221 len 24576 data: 5 7 6 6 5 6 16/30 Read block 188 len 8192 17/30 Write block 381 len 4096 data: 5 18/30 Write block 246 len 12288 data: 3 2 4 19/30 Read block 327 len 36864 20/30 Write block 569 len 8192 data: 7 8 21/30 Read block 353 len 20480 22/30 Write block 490 len 4096 data: 10 23/30 Write block 473 len 40960 data: 4 3 2 2 3 4 3 3 3 3 24/30 Write block 472 len 4096 data: 4 25/30 Write block 426 len 40960 data: 9 11 9 9 9 6 6 7 7 6 26/30 Write block 8 len 12288 data: 2 2 2 27/30 Read block 82 len 28672 28/30 Write block 522 len 32768 data: 4 7 7 6 7 6 6 5 29/30 Write block 123 len 12288 data: 8 9 9 30/30 Read block 242 len 36864 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:31 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1 Acked 1000 Write 9 Done Done Done false 2 Acked 1001 Write 3 Done Done Done false 3 Acked 1002 Read 9 Done Done Done false 4 Acked 1003 Read 8 Done Done Done false 5 Acked 1004 Read 10 Done Done Done false 6 Acked 1005 Write 3 Done Done Done false 7 Acked 1006 Write 10 Done Done Done false 8 Acked 1007 Read 9 Done Done Done false 9 Acked 1008 Write 7 Done Done Done false 10 Acked 1009 Read 6 Done Done Done false 11 Acked 1010 Write 1 Done Done Done false 12 Acked 1011 Read 2 Done Done Done false 13 Acked 1012 Write 4 Done Done Done false 14 Acked 1013 Read 9 Done Done Done false 15 Acked 1014 Write 6 Done Done Done false 16 Acked 1015 Read 2 Done Done Done false 17 Acked 1016 Write 1 Done Done Done false 18 Acked 1017 Write 3 Done Done Done false 19 Acked 1018 Read 9 Done Done Done false 20 Acked 1019 Write 2 Done Done Done false 21 Acked 1020 Read 5 Done Done Done false 22 Acked 1021 Write 1 Done Done Done false 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Write 1 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Flush 0 Done Done Sent false 29 Acked 1028 Write 8 Done Sent Sent false 30 Acked 1029 Write 3 Done Sent Sent false 31 Acked 1030 Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 3 4 7 Done 31 28 27 86 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1027 1027 0 Downstairs last five completed: Upstairs last five completed: 28 31 30 29 27 Wrote out file "/tmp/test_repair_verify.data" {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595534964Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595642754Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59567477Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595700038Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595723406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595744304Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595765281Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59578474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595803338Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59588513Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595904148Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595918016Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595934085Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595957032Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595970871Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59598322Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.595995539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596007767Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596019886Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596032045Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596044164Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596056353Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596068541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59608073Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596092769Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596104998Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596117096Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596129175Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596141504Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596156693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596169311Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59618157Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596193889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596206458Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596218846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596231375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596243734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596256373Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596268661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59628091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596293079Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596305228Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596318027Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596330405Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596342664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596355613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596370231Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59638267Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596394779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596407228Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596419566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596431805Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596443934Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596456143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596468332Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59648034Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596492439Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596504508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596516717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596528866Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596540945Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596553013Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596565292Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596580211Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596592799Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596605018Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596617167Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596629326Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596641535Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596653663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596665872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596678051Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59669016Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596702498Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596714817Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596727246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596739585Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596751954Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596764402Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596776771Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59679142Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596803779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596816937Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596829206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596842225Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596854303Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596866502Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596879881Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59689221Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596904509Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596916687Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596929846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596942115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596954224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596966322Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.596978461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59699069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597005668Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597018007Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597030206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597042375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597055454Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597067912Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597080181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.59709234Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597104439Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597116877Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597129456Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597142455Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597155953Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597168432Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597180651Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597208958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597222897Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597235346Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:28.597250294Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1433,"session_id":"1de913b8-76e5-4705-9484-69ab1605b5e8"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 118 118 118 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 58 58 58 113 113 113 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 61 61 61 117 117 117 F F F 6 120-139 59 59 59 115 115 115 T T T 7 140-159 62 62 62 118 118 118 F F F 8 160-179 60 60 60 116 116 116 F F F 9 180-199 62 62 62 118 118 118 F F F 10 200-219 60 60 60 116 116 116 F F F 11 220-239 62 62 62 118 118 118 F F F 12 240-259 62 62 62 118 118 118 F F F 13 260-279 62 62 62 118 118 118 F F F 14 280-299 62 62 62 118 118 118 F F F 15 300-319 60 60 60 116 116 116 F F F 16 320-339 58 58 58 112 112 112 F F F 17 340-359 62 62 62 118 118 118 F F F 18 360-379 62 62 62 118 118 118 F F F 19 380-399 62 62 62 118 118 118 F F F 20 400-419 54 54 54 104 104 104 F F F 21 420-439 62 62 62 118 118 118 F F F 22 440-459 61 61 61 117 117 117 F F F 23 460-479 62 62 62 118 118 118 F F F 24 480-499 62 62 62 118 118 118 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 58 58 58 113 113 113 T T T 27 540-559 60 60 60 116 116 116 F F F 28 560-579 62 62 62 118 118 118 F F F 29 580-599 58 58 58 114 114 114 F F F Max gen: 62, Max flush: 118 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 118 118 118 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 58 58 58 113 113 113 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 61 61 61 117 117 117 F F F 6 120-139 63 63 63 119 119 119 F F F 7 140-159 62 62 62 118 118 118 F F F 8 160-179 60 60 60 116 116 116 F F F 9 180-199 62 62 62 118 118 118 F F F 10 200-219 60 60 60 116 116 116 F F F 11 220-239 62 62 62 118 118 118 F F F 12 240-259 62 62 62 118 118 118 F F F 13 260-279 62 62 62 118 118 118 F F F 14 280-299 62 62 62 118 118 118 F F F 15 300-319 60 60 60 116 116 116 F F F 16 320-339 58 58 58 112 112 112 F F F 17 340-359 62 62 62 118 118 118 F F F 18 360-379 62 62 62 118 118 118 F F F 19 380-399 62 62 62 118 118 118 F F F 20 400-419 54 54 54 104 104 104 F F F 21 420-439 62 62 62 118 118 118 F F F 22 440-459 61 61 61 117 117 117 F F F 23 460-479 62 62 62 118 118 118 F F F 24 480-499 62 62 62 118 118 118 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 60 60 60 116 116 116 F F F 28 560-579 62 62 62 118 118 118 F F F 29 580-599 58 58 58 114 114 114 F F F Max gen: 63, Max flush: 119 Begin loop 32 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.784502108Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.784778661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.784794679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441} {"msg":"Crucible 74cd80d9-cfeb-440b-a9c5-904cf0bf0385 has session id: 5a316342-a39e-476d-ad18-c45630677f4b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785269002Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Upstairs opts: Upstairs UUID: 74cd80d9-cfeb-440b-a9c5-904cf0bf0385, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785302119Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Crucible stats registered with UUID: 74cd80d9-cfeb-440b-a9c5-904cf0bf0385","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785317368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785623777Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441} {"msg":"74cd80d9-cfeb-440b-a9c5-904cf0bf0385 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785704189Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785721667Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"0","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785736576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"0","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785757604Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785783701Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785809149Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"2","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785834076Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"2","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785857014Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"io task","client":"2","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785883991Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"io task","client":"0","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.785908109Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"io task","client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"ds_connection connected from Ok(127.0.0.1:36965)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.786053175Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"io task","client":"0","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:42.786093091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"io task","client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:42.786162224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"io task","client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:42.78619779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.786223098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.786247755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"ds_connection connected from Ok(127.0.0.1:49710)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.786269953Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"io task","client":"2","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.786292031Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"io task","client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.786791141Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"0","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.786818169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"0","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.786833067Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"2","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.786847146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"2","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.78741146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"0","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.787532888Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"2","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.787896352Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"0","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.787923659Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"0","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.787939677Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.787959146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"2","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.787973124Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"2","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:42.787986683Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.787429264Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"io task","client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"ds_connection connected from Ok(127.0.0.1:37122)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.787657761Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"io task","client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.788888009Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.788928945Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.789491299Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790019587Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790046464Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[0]R flush_numbers[0..12]: [118, 103, 113, 118, 110, 117, 119, 118, 116, 118, 116, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790066033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[0]R generation[0..12]: [62, 53, 58, 62, 57, 61, 63, 62, 60, 62, 60, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.79008882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790107138Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[1]R flush_numbers[0..12]: [118, 103, 113, 118, 110, 117, 119, 118, 116, 118, 116, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790124797Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[1]R generation[0..12]: [62, 53, 58, 62, 57, 61, 63, 62, 60, 62, 60, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790142025Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790159893Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[2]R flush_numbers[0..12]: [118, 103, 113, 118, 110, 117, 119, 118, 116, 118, 116, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790177282Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[2]R generation[0..12]: [62, 53, 58, 62, 57, 61, 63, 62, 60, 62, 60, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.79019458Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:35:52.790212368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790230716Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790247504Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Next flush: 120","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790264073Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790280571Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"":"downstairs","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.79029735Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790312738Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"0","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790332046Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"1","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790349114Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"client":"2","":"downstairs","session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790366033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790381451Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"74cd80d9-cfeb-440b-a9c5-904cf0bf0385 is now active with session: 5a316342-a39e-476d-ad18-c45630677f4b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.79039777Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790412718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441,"session_id":"5a316342-a39e-476d-ad18-c45630677f4b"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:35:52.790428127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 205 len 36864 data: 9 9 11 11 9 8 9 8 8 02/30 Read block 113 len 40960 03/30 Read block 205 len 8192 04/30 Read block 247 len 16384 05/30 Read block 307 len 20480 06/30 Read block 350 len 40960 07/30 Write block 468 len 12288 data: 7 7 6 08/30 Read block 350 len 16384 09/30 Read block 10 len 24576 10/30 Read block 19 len 16384 11/30 Read block 430 len 16384 12/30 Write block 175 len 36864 data: 5 5 5 4 4 5 5 4 4 13/30 Read block 129 len 4096 14/30 Read block 302 len 28672 15/30 Flush 16/30 Write block 55 len 16384 data: 6 8 8 8 17/30 Read block 47 len 24576 18/30 Read block 115 len 24576 19/30 Read block 88 len 16384 20/30 Read block 161 len 32768 21/30 Read block 518 len 40960 22/30 Write block 294 len 8192 data: 5 6 23/30 Read block 443 len 20480 24/30 Read block 110 len 12288 25/30 Write block 411 len 32768 data: 7 7 5 5 6 7 6 5 26/30 Read block 379 len 16384 27/30 Read block 104 len 12288 28/30 Write block 284 len 32768 data: 6 6 4 5 7 7 5 5 29/30 Write block 137 len 12288 data: 5 5 5 30/30 Write block 139 len 4096 data: 6 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Read 10 Done Done Done false 23 Acked 1022 Write 2 Done Done Done false 24 Acked 1023 Read 5 Done Done Done false 25 Acked 1024 Read 3 Done Done Done false 26 Acked 1025 Write 8 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Write 3 Sent Sent Sent false 31 Acked 1030 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 2 2 6 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:35:52.815350487Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1441} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 118 118 118 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 64 64 64 121 121 121 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 61 61 61 117 117 117 F F F 6 120-139 63 63 63 119 119 119 T T T 7 140-159 62 62 62 118 118 118 F F F 8 160-179 64 64 64 120 120 120 F F F 9 180-199 64 64 64 120 120 120 F F F 10 200-219 64 64 64 120 120 120 F F F 11 220-239 62 62 62 118 118 118 F F F 12 240-259 62 62 62 118 118 118 F F F 13 260-279 62 62 62 118 118 118 F F F 14 280-299 62 62 62 118 118 118 T T T 15 300-319 60 60 60 116 116 116 F F F 16 320-339 58 58 58 112 112 112 F F F 17 340-359 62 62 62 118 118 118 F F F 18 360-379 62 62 62 118 118 118 F F F 19 380-399 62 62 62 118 118 118 F F F 20 400-419 54 54 54 104 104 104 T T T 21 420-439 62 62 62 118 118 118 F F F 22 440-459 61 61 61 117 117 117 F F F 23 460-479 64 64 64 120 120 120 F F F 24 480-499 62 62 62 118 118 118 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 60 60 60 116 116 116 F F F 28 560-579 62 62 62 118 118 118 F F F 29 580-599 58 58 58 114 114 114 F F F Max gen: 64, Max flush: 121 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 118 118 118 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 64 64 64 121 121 121 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 61 61 61 117 117 117 F F F 6 120-139 65 65 65 122 122 122 F F F 7 140-159 62 62 62 118 118 118 F F F 8 160-179 64 64 64 120 120 120 F F F 9 180-199 64 64 64 120 120 120 F F F 10 200-219 64 64 64 120 120 120 F F F 11 220-239 62 62 62 118 118 118 F F F 12 240-259 62 62 62 118 118 118 F F F 13 260-279 62 62 62 118 118 118 F F F 14 280-299 65 65 65 122 122 122 F F F 15 300-319 60 60 60 116 116 116 F F F 16 320-339 58 58 58 112 112 112 F F F 17 340-359 62 62 62 118 118 118 F F F 18 360-379 62 62 62 118 118 118 F F F 19 380-399 62 62 62 118 118 118 F F F 20 400-419 65 65 65 122 122 122 F F F 21 420-439 62 62 62 118 118 118 F F F 22 440-459 61 61 61 117 117 117 F F F 23 460-479 64 64 64 120 120 120 F F F 24 480-499 62 62 62 118 118 118 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 60 60 60 116 116 116 F F F 28 560-579 62 62 62 118 118 118 F F F 29 580-599 58 58 58 114 114 114 F F F Max gen: 65, Max flush: 122 Begin loop 33 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.016435461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.01694547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.016978717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449} {"msg":"Crucible d4f5a349-0dde-49eb-8714-208b95e4fe6a has session id: ba4f2874-89da-4a13-bdfa-988b8c8397b9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017142821Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Upstairs opts: Upstairs UUID: d4f5a349-0dde-49eb-8714-208b95e4fe6a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017182127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Crucible stats registered with UUID: d4f5a349-0dde-49eb-8714-208b95e4fe6a","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017206264Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017519504Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449} {"msg":"d4f5a349-0dde-49eb-8714-208b95e4fe6a active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017615144Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017633672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017648461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017662149Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"1","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017675798Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"1","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017689297Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"2","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017712994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"2","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017740012Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"io task","client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017765719Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"io task","client":"1","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.017790417Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"io task","client":"2","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:43567)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.018004396Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"io task","client":"2","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:07.018181608Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"io task","client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:07.018221564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"io task","client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:51665)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.018249501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"io task","client":"1","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:07.018279598Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.018301056Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.018326533Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.018391837Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"io task","client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.018929484Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"1","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.01896738Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"1","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.018996657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"2","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.019022995Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"2","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.019616046Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"2","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.019654942Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"1","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.02018141Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"2","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.020209357Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"2","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.020229705Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.020252493Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"1","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.020268311Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"1","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:07.02028179Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.019759846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"io task","client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:62777)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.019870035Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"io task","client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.020915931Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.020947268Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.021522091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022107703Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022130201Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[0]R flush_numbers[0..12]: [118, 103, 121, 118, 110, 117, 122, 118, 120, 120, 120, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022149589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[0]R generation[0..12]: [62, 53, 64, 62, 57, 61, 65, 62, 64, 64, 64, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022164587Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022178976Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[1]R flush_numbers[0..12]: [118, 103, 121, 118, 110, 117, 122, 118, 120, 120, 120, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022193155Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[1]R generation[0..12]: [62, 53, 64, 62, 57, 61, 65, 62, 64, 64, 64, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022206993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022221242Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[2]R flush_numbers[0..12]: [118, 103, 121, 118, 110, 117, 122, 118, 120, 120, 120, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.02223511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[2]R generation[0..12]: [62, 53, 64, 62, 57, 61, 65, 62, 64, 64, 64, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022249149Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022263448Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"Wait for a query_work_queue command to finish before sending IO } {"msg":"Max found gen is 66","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022280206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022293634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Next flush: 123","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022306733Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022319792Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"":"downstairs","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.02233333Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022347579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"0","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022361178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"1","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022374526Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"client":"2","":"downstairs","session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022388255Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022400784Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"d4f5a349-0dde-49eb-8714-208b95e4fe6a is now active with session: ba4f2874-89da-4a13-bdfa-988b8c8397b9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022413793Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.022425751Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449,"session_id":"ba4f2874-89da-4a13-bdfa-988b8c8397b9"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:17.02243808Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 356 len 16384 data: 7 8 9 8 02/30 Flush 03/30 Write block 331 len 8192 data: 6 6 04/30 Read block 556 len 16384 05/30 Read block 466 len 4096 06/30 Write block 307 len 12288 data: 6 6 6 07/30 Read block 291 len 12288 08/30 Write block 109 len 40960 data: 4 3 5 6 6 8 8 6 6 4 09/30 Read block 322 len 8192 10/30 Read block 125 len 20480 11/30 Read block 563 len 12288 12/30 Write block 53 len 24576 data: 5 6 7 9 9 9 13/30 Read block 461 len 40960 14/30 Flush 15/30 Write block 288 len 24576 data: 8 8 6 6 5 4 16/30 Read block 79 len 20480 17/30 Read block 91 len 24576 18/30 Flush 19/30 Write block 9 len 36864 data: 3 3 3 4 5 7 7 8 8 20/30 Read block 443 len 16384 21/30 Read block 9 len 24576 22/30 Write block 438 len 28672 data: 6 4 7 8 8 8 9 23/30 Read block 143 len 32768 24/30 Read block 211 len 16384 25/30 Read block 220 len 28672 26/30 Write block 491 len 20480 data: 9 3 4 4 4 27/30 Write block 151 len 40960 data: 5 5 5 5 7 5 4 5 5 6 28/30 Write block 541 len 8192 data: 4 4 29/30 Read block 35 len 28672 30/30 Read block 486 len 28672 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 9 Done Done Done false 20 Acked 1019 Read 4 Done Done Done false 21 Acked 1020 Read 6 Done Done Done false 22 Acked 1021 Write 7 Done Done Done false 23 Acked 1022 Read 8 Done Done Done false 24 Acked 1023 Read 4 Done Done Done false 25 Acked 1024 Read 7 Done Done Done false 26 Acked 1025 Write 5 Done Done Done false 27 Acked 1026 Write 10 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 7 Done Done Done false 30 Acked 1029 Read 7 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 0 1 Done 12 11 12 35 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:17.049594099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1449} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 118 118 118 T T T 1 020-039 53 53 53 103 103 103 F F F 2 040-059 66 66 66 124 124 124 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 66 66 66 124 124 124 F F F 6 120-139 65 65 65 122 122 122 F F F 7 140-159 62 62 62 118 118 118 T T T 8 160-179 64 64 64 120 120 120 T T T 9 180-199 64 64 64 120 120 120 F F F 10 200-219 64 64 64 120 120 120 F F F 11 220-239 62 62 62 118 118 118 F F F 12 240-259 62 62 62 118 118 118 F F F 13 260-279 62 62 62 118 118 118 F F F 14 280-299 66 66 66 125 125 125 F F F 15 300-319 66 66 66 124 124 124 F F F 16 320-339 66 66 66 124 124 124 F F F 17 340-359 66 66 66 123 123 123 F F F 18 360-379 62 62 62 118 118 118 F F F 19 380-399 62 62 62 118 118 118 F F F 20 400-419 65 65 65 122 122 122 F F F 21 420-439 62 62 62 118 118 118 T T T 22 440-459 61 61 61 117 117 117 T T T 23 460-479 64 64 64 120 120 120 F F F 24 480-499 62 62 62 118 118 118 T T T 25 500-519 58 58 58 111 111 111 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 60 60 60 116 116 116 T T T 28 560-579 62 62 62 118 118 118 F F F 29 580-599 58 58 58 114 114 114 F F F Max gen: 66, Max flush: 125 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 67 67 67 126 126 126 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 66 66 66 124 124 124 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 66 66 66 124 124 124 F F F 6 120-139 65 65 65 122 122 122 F F F 7 140-159 67 67 67 126 126 126 F F F 8 160-179 67 67 67 126 126 126 F F F 9 180-199 64 64 64 120 120 120 F F F 10 200-219 64 64 64 120 120 120 F F F 11 220-239 62 62 62 118 118 118 F F F 12 240-259 62 62 62 118 118 118 F F F 13 260-279 62 62 62 118 118 118 F F F 14 280-299 66 66 66 125 125 125 F F F 15 300-319 66 66 66 124 124 124 F F F 16 320-339 66 66 66 124 124 124 F F F 17 340-359 66 66 66 123 123 123 F F F 18 360-379 62 62 62 118 118 118 F F F 19 380-399 62 62 62 118 118 118 F F F 20 400-419 65 65 65 122 122 122 F F F 21 420-439 67 67 67 126 126 126 F F F 22 440-459 67 67 67 126 126 126 F F F 23 460-479 64 64 64 120 120 120 F F F 24 480-499 67 67 67 126 126 126 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 67 67 67 126 126 126 F F F 28 560-579 62 62 62 118 118 118 F F F 29 580-599 58 58 58 114 114 114 F F F Max gen: 67, Max flush: 126 Begin loop 34 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.302528324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.302851212Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.302870779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457} {"msg":"Crucible a394bfb4-e499-4a98-8158-68540c84af15 has session id: d721109b-5525-40a7-af2c-2a21a100245b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303348562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Upstairs opts: Upstairs UUID: a394bfb4-e499-4a98-8158-68540c84af15, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303381459Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Crucible stats registered with UUID: a394bfb4-e499-4a98-8158-68540c84af15","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303396927Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303695508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457} {"msg":"a394bfb4-e499-4a98-8158-68540c84af15 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303791178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303815966Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303832344Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303846453Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"1","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303860481Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"1","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.30387765Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"2","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303891898Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"2","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303909477Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"io task","client":"2","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303923475Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"io task","client":"1","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.303937224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"io task","client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:31.304150473Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"io task","client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:31.304192479Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"io task","client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:31.304221466Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.304361272Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.304397578Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"ds_connection connected from Ok(127.0.0.1:56895)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.304423686Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"io task","client":"2","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"ds_connection connected from Ok(127.0.0.1:63168)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.304450233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"io task","client":"1","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.304472631Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"io task","client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.305021306Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"1","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.305045584Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"1","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.305063472Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"2","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.305077561Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"2","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.305649324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"1","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.305686891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"2","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.306182351Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"1","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.306208099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"1","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.306222447Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.306250295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"2","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.306264663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"2","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:31.306277932Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.305566958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"io task","client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"ds_connection connected from Ok(127.0.0.1:40664)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.305695715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"io task","client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.307155891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.307184998Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.30766564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308186709Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308207717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[0]R flush_numbers[0..12]: [126, 103, 124, 118, 110, 124, 122, 126, 126, 120, 120, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308222005Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[0]R generation[0..12]: [67, 53, 66, 62, 57, 66, 65, 67, 67, 64, 64, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308236404Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308250952Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[1]R flush_numbers[0..12]: [126, 103, 124, 118, 110, 124, 122, 126, 126, 120, 120, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308265021Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[1]R generation[0..12]: [67, 53, 66, 62, 57, 66, 65, 67, 67, 64, 64, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308278839Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308293088Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[2]R flush_numbers[0..12]: [126, 103, 124, 118, 110, 124, 122, 126, 126, 120, 120, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308307227Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[2]R generation[0..12]: [67, 53, 66, 62, 57, 66, 65, 67, 67, 64, 64, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308321155Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308335444Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308348593Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"Wait for a query_work_queue command to finish before sending IO msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308364791Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Next flush: 127","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.30837803Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308394408Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"":"downstairs","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308407927Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308419885Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"0","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308433074Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"1","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308446393Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"client":"2","":"downstairs","session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308459912Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.3084723Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"a394bfb4-e499-4a98-8158-68540c84af15 is now active with session: d721109b-5525-40a7-af2c-2a21a100245b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308485389Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308497468Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457,"session_id":"d721109b-5525-40a7-af2c-2a21a100245b"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:41.308509717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 108 len 28672 data: 5 5 4 6 7 7 9 02/30 Read block 206 len 36864 03/30 Read block 337 len 32768 04/30 Read block 58 len 24576 05/30 Read block 209 len 8192 06/30 Write block 7 len 32768 data: 2 3 4 4 4 5 6 8 07/30 Write block 575 len 24576 data: 9 10 8 4 4 4 08/30 Write block 183 len 36864 data: 5 3 3 3 4 3 5 8 9 09/30 Read block 32 len 36864 10/30 Read block 582 len 40960 11/30 Read block 575 len 4096 12/30 Flush 13/30 Write block 289 len 32768 data: 9 7 7 6 5 6 7 7 14/30 Read block 523 len 28672 15/30 Write block 260 len 32768 data: 3 3 5 5 5 5 5 4 16/30 Write block 418 len 16384 data: 6 6 8 8 17/30 Read block 380 len 40960 18/30 Write block 51 len 20480 data: 3 3 6 7 8 19/30 Read block 130 len 20480 20/30 Read block 498 len 4096 21/30 Write block 483 len 12288 data: 4 4 5 22/30 Read block 292 len 12288 23/30 Flush 24/30 Write block 594 len 4096 data: 9 25/30 Write block 176 len 12288 data: 6 6 5 26/30 Write block 584 len 36864 data: 7 7 7 6 5 7 7 7 6 27/30 Write block 386 len 20480 data: 8 8 10 12 11 28/30 Write block 331 len 12288 data: 7 7 5 29/30 Write block 598 len 8192 data: 5 6 30/30 Read block 444 len 8192 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 1 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Write 9 Done Done Done false 28 Acked 1027 Write 5 Done Done Done false 29 Acked 1028 Write 3 Done Done Done false 30 Acked 1029 Write 2 Done Done Done false 31 Acked 1030 Read 2 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:41.333819268Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1457} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 127 127 127 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 68 68 68 128 128 128 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 68 68 68 127 127 127 F F F 6 120-139 65 65 65 122 122 122 F F F 7 140-159 67 67 67 126 126 126 F F F 8 160-179 67 67 67 126 126 126 T T T 9 180-199 68 68 68 127 127 127 F F F 10 200-219 64 64 64 120 120 120 F F F 11 220-239 62 62 62 118 118 118 F F F 12 240-259 62 62 62 118 118 118 F F F 13 260-279 68 68 68 128 128 128 F F F 14 280-299 68 68 68 128 128 128 F F F 15 300-319 66 66 66 124 124 124 F F F 16 320-339 66 66 66 124 124 124 T T T 17 340-359 66 66 66 123 123 123 F F F 18 360-379 62 62 62 118 118 118 F F F 19 380-399 62 62 62 118 118 118 T T T 20 400-419 68 68 68 128 128 128 F F F 21 420-439 68 68 68 128 128 128 F F F 22 440-459 67 67 67 126 126 126 F F F 23 460-479 64 64 64 120 120 120 F F F 24 480-499 68 68 68 128 128 128 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 67 67 67 126 126 126 F F F 28 560-579 68 68 68 127 127 127 F F F 29 580-599 68 68 68 127 127 127 T T T Max gen: 68, Max flush: 128 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 127 127 127 F F F 1 020-039 53 53 53 103 103 103 F F F 2 040-059 68 68 68 128 128 128 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 68 68 68 127 127 127 F F F 6 120-139 65 65 65 122 122 122 F F F 7 140-159 67 67 67 126 126 126 F F F 8 160-179 69 69 69 129 129 129 F F F 9 180-199 68 68 68 127 127 127 F F F 10 200-219 64 64 64 120 120 120 F F F 11 220-239 62 62 62 118 118 118 F F F 12 240-259 62 62 62 118 118 118 F F F 13 260-279 68 68 68 128 128 128 F F F 14 280-299 68 68 68 128 128 128 F F F 15 300-319 66 66 66 124 124 124 F F F 16 320-339 69 69 69 129 129 129 F F F 17 340-359 66 66 66 123 123 123 F F F 18 360-379 62 62 62 118 118 118 F F F 19 380-399 69 69 69 129 129 129 F F F 20 400-419 68 68 68 128 128 128 F F F 21 420-439 68 68 68 128 128 128 F F F 22 440-459 67 67 67 126 126 126 F F F 23 460-479 64 64 64 120 120 120 F F F 24 480-499 68 68 68 128 128 128 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 67 67 67 126 126 126 F F F 28 560-579 68 68 68 127 127 127 F F F 29 580-599 69 69 69 129 129 129 F F F Max gen: 69, Max flush: 129 Begin loop 35 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.546540599Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.546827471Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.546845089Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465} {"msg":"Crucible ca999aab-3a18-426f-93b9-0e5954a408fc has session id: a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547234311Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Upstairs opts: Upstairs UUID: ca999aab-3a18-426f-93b9-0e5954a408fc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547279066Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Crucible stats registered with UUID: ca999aab-3a18-426f-93b9-0e5954a408fc","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547297114Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.54754313Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465} {"msg":"ca999aab-3a18-426f-93b9-0e5954a408fc active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547660658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547678547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547692785Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547713233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"1","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547738251Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"1","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547769488Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"2","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547793845Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"2","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547817073Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"io task","client":"2","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.54784137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"io task","client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.547864978Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"io task","client":"1","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ds_connection connected from Ok(127.0.0.1:32901)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.548060749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"io task","client":"2","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ds_connection connected from Ok(127.0.0.1:36983)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.548099365Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"io task","client":"1","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:55.548259089Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"io task","client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:55.548299595Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"io task","client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:36:55.548329622Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.54835559Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.548379517Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.548470548Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"io task","client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.548994136Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"1","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.549031632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"1","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.549064539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"2","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.549093516Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"2","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.54966329Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"2","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.549704536Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"1","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.55026861Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"2","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.550295777Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"2","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.550310966Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.550323684Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"1","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.550337463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"1","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:36:55.550351232Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.549193913Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"io task","client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ds_connection connected from Ok(127.0.0.1:47748)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.549401843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"io task","client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.550618522Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.550659428Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.55124269Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551786636Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551805824Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[0]R flush_numbers[0..12]: [127, 103, 128, 118, 110, 127, 122, 126, 129, 127, 120, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551820723Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[0]R generation[0..12]: [68, 53, 68, 62, 57, 68, 65, 67, 69, 68, 64, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551834922Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.55184923Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[1]R flush_numbers[0..12]: [127, 103, 128, 118, 110, 127, 122, 126, 129, 127, 120, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551863219Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[1]R generation[0..12]: [68, 53, 68, 62, 57, 68, 65, 67, 69, 68, 64, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551877077Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551891366Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[2]R flush_numbers[0..12]: [127, 103, 128, 118, 110, 127, 122, 126, 129, 127, 120, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551905455Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[2]R generation[0..12]: [68, 53, 68, 62, 57, 68, 65, 67, 69, 68, 64, 62]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551919243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551933452Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.5519465Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pidWait for a query_work_queue command to finish before sending IO ":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551963229Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Next flush: 130","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551976197Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.551992496Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"":"downstairs","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.552006014Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.552018083Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"0","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.552031272Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"1","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.552044541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"client":"2","":"downstairs","session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.552058049Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.552070428Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"ca999aab-3a18-426f-93b9-0e5954a408fc is now active with session: a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.552083347Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.552095406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465,"session_id":"a4e5c92c-b9c4-4ec8-9f9e-5cfeb7488384"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:05.552107615Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 165 len 28672 data: 4 4 3 2 3 3 4 02/30 Read block 355 len 36864 03/30 Read block 232 len 4096 04/30 Write block 104 len 40960 data: 7 6 6 5 6 6 5 7 8 8 05/30 Read block 333 len 12288 06/30 Read block 498 len 28672 07/30 Read block 93 len 32768 08/30 Read block 345 len 24576 09/30 Write block 37 len 4096 data: 11 10/30 Write block 177 len 4096 data: 7 11/30 Read block 334 len 20480 12/30 Write block 45 len 20480 data: 4 4 4 3 3 13/30 Read block 162 len 36864 14/30 Read block 337 len 16384 15/30 Read block 534 len 4096 16/30 Write block 317 len 36864 data: 6 6 7 8 8 6 6 6 7 17/30 Read block 355 len 24576 18/30 Read block 0 len 32768 19/30 Read block 11 len 32768 20/30 Write block 30 len 12288 data: 9 9 9 21/30 Read block 196 len 8192 22/30 Flush 23/30 Write block 373 len 16384 data: 4 5 5 4 24/30 Write block 438 len 20480 data: 7 5 8 9 9 25/30 Write block 576 len 36864 data: 11 9 5 5 5 5 6 5 8 26/30 Read block 392 len 16384 27/30 Read block 531 len 40960 28/30 Write block 550 len 12288 data: 8 9 8 29/30 Write block 236 len 36864 data: 6 4 4 4 5 3 3 3 3 30/30 Write block 554 len 28672 data: 6 5 7 7 8 7 7 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 4 Done Done Done false 24 Acked 1023 Write 5 Done Done Done false 25 Acked 1024 Write 9 Done Done Done false 26 Acked 1025 Read 4 Done Done Done false 27 Acked 1026 Read 10 Done Done Done false 28 Acked 1027 Write 3 Done Done Done false 29 Acked 1028 Write 9 Done Done Done false 30 Acked 1029 Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:37:05.576630944Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1465} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 127 127 127 F F F 1 020-039 70 70 70 130 130 130 F F F 2 040-059 70 70 70 130 130 130 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 70 70 70 130 130 130 F F F 6 120-139 65 65 65 122 122 122 F F F 7 140-159 67 67 67 126 126 126 F F F 8 160-179 70 70 70 130 130 130 F F F 9 180-199 68 68 68 127 127 127 F F F 10 200-219 64 64 64 120 120 120 F F F 11 220-239 62 62 62 118 118 118 T T T 12 240-259 62 62 62 118 118 118 T T T 13 260-279 68 68 68 128 128 128 F F F 14 280-299 68 68 68 128 128 128 F F F 15 300-319 70 70 70 130 130 130 F F F 16 320-339 70 70 70 130 130 130 F F F 17 340-359 66 66 66 123 123 123 F F F 18 360-379 62 62 62 118 118 118 T T T 19 380-399 69 69 69 129 129 129 F F F 20 400-419 68 68 68 128 128 128 F F F 21 420-439 68 68 68 128 128 128 T T T 22 440-459 67 67 67 126 126 126 T T T 23 460-479 64 64 64 120 120 120 F F F 24 480-499 68 68 68 128 128 128 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 67 67 67 126 126 126 T T T 28 560-579 68 68 68 127 127 127 T T T 29 580-599 69 69 69 129 129 129 T T T Max gen: 70, Max flush: 130 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 127 127 127 F F F 1 020-039 70 70 70 130 130 130 F F F 2 040-059 70 70 70 130 130 130 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 70 70 70 130 130 130 F F F 6 120-139 65 65 65 122 122 122 F F F 7 140-159 67 67 67 126 126 126 F F F 8 160-179 70 70 70 130 130 130 F F F 9 180-199 68 68 68 127 127 127 F F F 10 200-219 64 64 64 120 120 120 F F F 11 220-239 71 71 71 131 131 131 F F F 12 240-259 71 71 71 131 131 131 F F F 13 260-279 68 68 68 128 128 128 F F F 14 280-299 68 68 68 128 128 128 F F F 15 300-319 70 70 70 130 130 130 F F F 16 320-339 70 70 70 130 130 130 F F F 17 340-359 66 66 66 123 123 123 F F F 18 360-379 71 71 71 131 131 131 F F F 19 380-399 69 69 69 129 129 129 F F F 20 400-419 68 68 68 128 128 128 F F F 21 420-439 71 71 71 131 131 131 F F F 22 440-459 71 71 71 131 131 131 F F F 23 460-479 64 64 64 120 120 120 F F F 24 480-499 68 68 68 128 128 128 F F F 25 500-519 58 58 58 111 111 111 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 71 71 71 131 131 131 F F F 28 560-579 71 71 71 131 131 131 F F F 29 580-599 71 71 71 131 131 131 F F F Max gen: 71, Max flush: 131 Begin loop 36 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.841383908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.841727114Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.841750482Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473} {"msg":"Crucible 4eb31fab-ae82-435a-a5f6-8fd9cdf19d54 has session id: 93f60506-bb18-493d-9594-14d46b023fd6","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842028674Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Upstairs opts: Upstairs UUID: 4eb31fab-ae82-435a-a5f6-8fd9cdf19d54, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.84207531Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Crucible stats registered with UUID: 4eb31fab-ae82-435a-a5f6-8fd9cdf19d54","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842097348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842392198Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473} {"msg":"4eb31fab-ae82-435a-a5f6-8fd9cdf19d54 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842429325Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842443683Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"0","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842458392Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"0","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842472271Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"1","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842500288Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"1","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842526155Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842553632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.84257939Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"io task","client":"0","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842605117Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"io task","client":"1","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842629315Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"io task","client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"ds_connection connected from Ok(127.0.0.1:55045)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842728065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"io task","client":"0","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:37:19.842764522Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"io task","client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:37:19.842792599Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"io task","client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:37:19.842819396Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842833685Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842855602Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"ds_connection connected from Ok(127.0.0.1:53713)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.842935634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"io task","client":"1","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.843215207Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"io task","client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.843730886Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"1","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.843769192Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"1","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.843792629Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"0","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.843814757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"0","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.844440255Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"1","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.844469033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"0","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.845051025Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"0","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.845085112Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"0","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.845108289Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.845127817Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"1","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.845148765Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"1","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:19.845170313Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.844425081Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"io task","client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"ds_connection connected from Ok(127.0.0.1:40530)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.844827111Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"io task","client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.846135202Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.846162199Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.84675598Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847308495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847327233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[0]R flush_numbers[0..12]: [127, 130, 130, 118, 110, 130, 122, 126, 130, 127, 120, 131]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847353231Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[0]R generation[0..12]: [68, 70, 70, 62, 57, 70, 65, 67, 70, 68, 64, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847372259Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847390167Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[1]R flush_numbers[0..12]: [127, 130, 130, 118, 110, 130, 122, 126, 130, 127, 120, 131]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847407965Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[1]R generation[0..12]: [68, 70, 70, 62, 57, 70, 65, 67, 70, 68, 64, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847425324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847442812Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[2]R flush_numbers[0..12]: [127, 130, 130, 118, 110, 130, 122, 126, 130, 127, 120, 131]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.84746007Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[2]R generation[0..12]: [68, 70, 70, 62, 57, 70, 65, 67, 70, 68, 64, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847477059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847494517Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847514925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847531473Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Next flush: 132","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847552691Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.84756968Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"":"downstairs","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847586328Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847601316Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"0","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847618045Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"1","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847634363Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"client":"2","":"downstairs","session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847650781Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.84766602Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"4eb31fab-ae82-435a-a5f6-8fd9cdf19d54 is now active with session: 93f60506-bb18-493d-9594-14d46b023fd6","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847682028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847697107Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473,"session_id":"93f60506-bb18-493d-9594-14d46b023fd6"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:29.847712425Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 139 len 20480 data: 7 5 4 5 6 02/30 Read block 0 len 24576 03/30 Write block 191 len 40960 data: 10 10 10 10 9 6 8 9 9 8 04/30 Read block 181 len 40960 05/30 Read block 0 len 12288 06/30 Read block 238 len 4096 07/30 Write block 349 len 20480 data: 5 4 3 3 5 08/30 Write block 201 len 28672 data: 7 7 7 8 10 10 12 09/30 Read block 297 len 12288 10/30 Write block 121 len 36864 data: 4 5 9 10 10 10 7 4 3 11/30 Read block 397 len 16384 12/30 Write block 140 len 28672 data: 6 5 6 7 5 4 5 13/30 Write block 12 len 28672 data: 6 7 9 8 9 9 7 14/30 Write block 512 len 4096 data: 7 15/30 Read block 169 len 12288 16/30 Flush 17/30 Flush 18/30 Write block 177 len 20480 data: 8 6 5 6 6 19/30 Read block 455 len 20480 20/30 Read block 222 len 24576 21/30 Read block 63 len 24576 22/30 Write block 309 len 24576 data: 7 6 6 6 6 7 23/30 Read block 111 len 20480 24/30 Read block 312 len 28672 25/30 Write block 42 len 32768 data: 4 6 4 5 5 5 4 4 26/30 Read block 493 len 8192 27/30 Write block 581 len 8192 data: 6 7 28/30 Write block 375 len 8192 data: 6 5 29/30 Read block 57 len 16384 30/30 Write block 470 len 20480 data: 7 5 5 5 4 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 5 Done Done Done false 19 Acked 1018 Read 5 Done Done Done false 20 Acked 1019 Read 6 Done Done Done false 21 Acked 1020 Read 6 Done Done Done false 22 Acked 1021 Write 6 Done Done Done false 23 Acked 1022 Read 5 Done Done Done false 24 Acked 1023 Read 7 Done Done Done false 25 Acked 1024 Write 8 Done Done Done false 26 Acked 1025 Read 2 Done Done Done false 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 4 Done Done Done false 30 Acked 1029 Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 12 12 12 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:37:29.871622326Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1473} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 132 132 132 F F F 1 020-039 70 70 70 130 130 130 F F F 2 040-059 70 70 70 130 130 130 T T T 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 70 70 70 130 130 130 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 72 72 72 132 132 132 F F F 8 160-179 70 70 70 130 130 130 T T T 9 180-199 72 72 72 132 132 132 T T T 10 200-219 72 72 72 132 132 132 F F F 11 220-239 71 71 71 131 131 131 F F F 12 240-259 71 71 71 131 131 131 F F F 13 260-279 68 68 68 128 128 128 F F F 14 280-299 68 68 68 128 128 128 F F F 15 300-319 70 70 70 130 130 130 T T T 16 320-339 70 70 70 130 130 130 F F F 17 340-359 72 72 72 132 132 132 F F F 18 360-379 71 71 71 131 131 131 T T T 19 380-399 69 69 69 129 129 129 F F F 20 400-419 68 68 68 128 128 128 F F F 21 420-439 71 71 71 131 131 131 F F F 22 440-459 71 71 71 131 131 131 F F F 23 460-479 64 64 64 120 120 120 T T T 24 480-499 68 68 68 128 128 128 F F F 25 500-519 72 72 72 132 132 132 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 71 71 71 131 131 131 F F F 28 560-579 71 71 71 131 131 131 F F F 29 580-599 71 71 71 131 131 131 T T T Max gen: 72, Max flush: 132 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 132 132 132 F F F 1 020-039 70 70 70 130 130 130 F F F 2 040-059 73 73 73 133 133 133 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 F F F 5 100-119 70 70 70 130 130 130 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 72 72 72 132 132 132 F F F 8 160-179 73 73 73 133 133 133 F F F 9 180-199 73 73 73 133 133 133 F F F 10 200-219 72 72 72 132 132 132 F F F 11 220-239 71 71 71 131 131 131 F F F 12 240-259 71 71 71 131 131 131 F F F 13 260-279 68 68 68 128 128 128 F F F 14 280-299 68 68 68 128 128 128 F F F 15 300-319 73 73 73 133 133 133 F F F 16 320-339 70 70 70 130 130 130 F F F 17 340-359 72 72 72 132 132 132 F F F 18 360-379 73 73 73 133 133 133 F F F 19 380-399 69 69 69 129 129 129 F F F 20 400-419 68 68 68 128 128 128 F F F 21 420-439 71 71 71 131 131 131 F F F 22 440-459 71 71 71 131 131 131 F F F 23 460-479 73 73 73 133 133 133 F F F 24 480-499 68 68 68 128 128 128 F F F 25 500-519 72 72 72 132 132 132 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 71 71 71 131 131 131 F F F 28 560-579 71 71 71 131 131 131 F F F 29 580-599 73 73 73 133 133 133 F F F Max gen: 73, Max flush: 133 Begin loop 37 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.129973702Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.130410819Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.130439556Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481} {"msg":"Crucible b3cc485e-175e-4a7f-af4d-5cd054bbf00a has session id: b40b0e2b-9a87-4757-a5aa-3e1779e80551","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.130740996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Upstairs opts: Upstairs UUID: b3cc485e-175e-4a7f-af4d-5cd054bbf00a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.130893681Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Crucible stats registered with UUID: b3cc485e-175e-4a7f-af4d-5cd054bbf00a","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.130926008Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131240547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481} {"msg":"b3cc485e-175e-4a7f-af4d-5cd054bbf00a active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131270894Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131295111Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131324378Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131347206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"1","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131371364Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"1","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131396971Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"2","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131421499Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"2","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131443397Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"io task","client":"2","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131467564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"io task","client":"1","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131492652Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"io task","client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"ds_connection connected from Ok(127.0.0.1:52464)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131598581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"io task","client":"2","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"ds_connection connected from Ok(127.0.0.1:43924)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131779643Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"io task","client":"1","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:37:44.131818659Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"io task","client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:37:44.131837567Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"io task","client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:37:44.131908211Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131925959Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.131939737Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.132029999Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"io task","client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.132423919Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"1","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.132461306Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"1","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.132489343Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"2","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.13251617Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"2","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.133074055Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"2","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.133100802Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"1","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.133644738Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"2","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.133683104Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"2","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.133711722Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.133750108Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"1","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.133779535Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"1","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:44.133796153Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.133552654Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"io task","client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"ds_connection connected from Ok(127.0.0.1:63670)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.133872672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"io task","client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.134780172Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.134807189Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.135387412Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136052006Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136084483Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[0]R flush_numbers[0..12]: [132, 130, 133, 118, 110, 130, 132, 132, 133, 133, 132, 131]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136104541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[0]R generation[0..12]: [72, 70, 73, 62, 57, 70, 72, 72, 73, 73, 72, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136122829Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136141107Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[1]R flush_numbers[0..12]: [132, 130, 133, 118, 110, 130, 132, 132, 133, 133, 132, 131]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136159055Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[1]R generation[0..12]: [72, 70, 73, 62, 57, 70, 72, 72, 73, 73, 72, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136176384Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136194802Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[2]R flush_numbers[0..12]: [132, 130, 133, 118, 110, 130, 132, 132, 133, 133, 132, 131]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:37:54.13621215Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[2]R generation[0..12]: [72, 70, 73, 62, 57, 70, 72, 72, 73, 73, 72, 71]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136229758Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136247537Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136267844Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136284723Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Next flush: 134","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136301051Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.13631726Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"":"downstairs","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136334128Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136349536Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"0","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136366645Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"1","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136383563Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"client":"2","":"downstairs","session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136400521Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.13641613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"b3cc485e-175e-4a7f-af4d-5cd054bbf00a is now active with session: b40b0e2b-9a87-4757-a5aa-3e1779e80551","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136432368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:37:54.136447237Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481,"session_id":"b40b0e2b-9a87-4757-a5aa-3e1779e80551"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2024-07-09T21:37:54.136462485Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 425 len 12288 data: 9 10 12 03/30 Flush 04/30 Write block 102 len 36864 data: 7 8 8 7 7 6 7 7 6 05/30 Read block 283 len 28672 06/30 Flush 07/30 Write block 391 len 40960 data: 14 13 12 11 10 7 5 2 4 3 08/30 Write block 299 len 32768 data: 7 8 7 5 5 6 8 8 09/30 Read block 457 len 16384 10/30 Read block 222 len 36864 11/30 Write block 265 len 32768 data: 6 6 5 3 5 4 4 3 12/30 Write block 238 len 4096 data: 5 13/30 Write block 557 len 8192 data: 8 9 14/30 Read block 363 len 40960 15/30 Write block 325 len 16384 data: 8 7 7 7 16/30 Flush 17/30 Write block 483 len 16384 data: 5 5 6 7 18/30 Read block 51 len 36864 19/30 Read block 401 len 4096 20/30 Write block 548 len 32768 data: 7 7 9 10 9 6 7 6 21/30 Write block 97 len 12288 data: 9 8 8 22/30 Write block 312 len 24576 data: 7 7 8 5 5 7 23/30 Write block 563 len 36864 data: 9 9 9 8 6 10 8 9 11 24/30 Write block 166 len 8192 data: 5 4 25/30 Read block 183 len 4096 26/30 Write block 593 len 28672 data: 7 10 9 8 6 6 7 27/30 Read block 583 len 16384 28/30 Read block 547 len 12288 29/30 Write block 460 len 32768 data: 7 7 10 9 7 7 7 7 30/30 Read block 473 len 32768 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 4 Done Done Done false 19 Acked 1018 Read 9 Done Done Done false 20 Acked 1019 Read 1 Done Done Done false 21 Acked 1020 Write 8 Done Done Done false 22 Acked 1021 Write 3 Done Done Done false 23 Acked 1022 Write 6 Done Done Done false 24 Acked 1023 Write 9 Done Done Done false 25 Acked 1024 Write 2 Done Done Done false 26 Acked 1025 Read 1 Done Done Done false 27 Acked 1026 Write 7 Done Done Done false 28 Acked 1027 Read 4 Done Done Done false 29 Acked 1028 Read 3 Done Done Done false 30 Acked 1029 Write 8 Done Done Done false 31 Acked 1030 Read 8 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 1 2 Done 13 14 13 40 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:37:54.164420854Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1481} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 132 132 132 F F F 1 020-039 70 70 70 130 130 130 F F F 2 040-059 73 73 73 133 133 133 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 57 57 57 110 110 110 T T T 5 100-119 74 74 74 136 136 136 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 72 72 72 132 132 132 F F F 8 160-179 73 73 73 133 133 133 T T T 9 180-199 73 73 73 133 133 133 F F F 10 200-219 72 72 72 132 132 132 F F F 11 220-239 74 74 74 137 137 137 F F F 12 240-259 71 71 71 131 131 131 F F F 13 260-279 74 74 74 137 137 137 F F F 14 280-299 74 74 74 137 137 137 F F F 15 300-319 74 74 74 137 137 137 T T T 16 320-339 74 74 74 138 138 138 F F F 17 340-359 72 72 72 132 132 132 F F F 18 360-379 73 73 73 133 133 133 F F F 19 380-399 74 74 74 137 137 137 F F F 20 400-419 74 74 74 137 137 137 F F F 21 420-439 74 74 74 135 135 135 F F F 22 440-459 71 71 71 131 131 131 F F F 23 460-479 73 73 73 133 133 133 T T T 24 480-499 68 68 68 128 128 128 T T T 25 500-519 72 72 72 132 132 132 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 74 74 74 137 137 137 T T T 28 560-579 71 71 71 131 131 131 T T T 29 580-599 73 73 73 133 133 133 T T T Max gen: 74, Max flush: 138 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 132 132 132 F F F 1 020-039 70 70 70 130 130 130 F F F 2 040-059 73 73 73 133 133 133 F F F 3 060-079 62 62 62 118 118 118 F F F 4 080-099 75 75 75 139 139 139 F F F 5 100-119 74 74 74 136 136 136 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 72 72 72 132 132 132 F F F 8 160-179 75 75 75 139 139 139 F F F 9 180-199 73 73 73 133 133 133 F F F 10 200-219 72 72 72 132 132 132 F F F 11 220-239 74 74 74 137 137 137 F F F 12 240-259 71 71 71 131 131 131 F F F 13 260-279 74 74 74 137 137 137 F F F 14 280-299 74 74 74 137 137 137 F F F 15 300-319 75 75 75 139 139 139 F F F 16 320-339 74 74 74 138 138 138 F F F 17 340-359 72 72 72 132 132 132 F F F 18 360-379 73 73 73 133 133 133 F F F 19 380-399 74 74 74 137 137 137 F F F 20 400-419 74 74 74 137 137 137 F F F 21 420-439 74 74 74 135 135 135 F F F 22 440-459 71 71 71 131 131 131 F F F 23 460-479 75 75 75 139 139 139 F F F 24 480-499 75 75 75 139 139 139 F F F 25 500-519 72 72 72 132 132 132 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 75 75 75 139 139 139 F F F 28 560-579 75 75 75 139 139 139 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 75, Max flush: 139 Begin loop 38 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.4380233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.438643428Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.438678565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489} {"msg":"Crucible 41d72f86-0523-4547-9116-7ed945ee74cb has session id: 4a2cbfdd-da1b-4baf-9a6f-ad403a41f312","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.438700992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Upstairs opts: Upstairs UUID: 41d72f86-0523-4547-9116-7ed945ee74cb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.43872817Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Crucible stats registered with UUID: 41d72f86-0523-4547-9116-7ed945ee74cb","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.438742908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439097273Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489} {"msg":"41d72f86-0523-4547-9116-7ed945ee74cb active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.43913145Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439156097Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"0","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439182105Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"0","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439204592Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"1","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.43922762Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"1","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439252748Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439278985Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439305142Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"io task","client":"0","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.43932975Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"io task","client":"1","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439353648Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"io task","client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"ds_connection connected from Ok(127.0.0.1:48682)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439523171Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"io task","client":"0","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"ds_connection connected from Ok(127.0.0.1:54976)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439561057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"io task","client":"1","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:08.439767427Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"io task","client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:08.439809512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"io task","client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:08.43983445Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439852848Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439870437Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.439928131Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"io task","client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.440366037Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"0","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.440390985Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"0","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.440406193Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"1","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.440429941Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"1","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.441037421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"1","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.441064418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"0","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.44155443Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"0","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.441581277Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"0","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.441595926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.441621843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"1","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.441636981Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"1","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:08.44165028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.44130901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"io task","client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"ds_connection connected from Ok(127.0.0.1:64880)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.44160536Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"io task","client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.442731589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.442755926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443269925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443805322Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.44382716Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[0]R flush_numbers[0..12]: [132, 130, 133, 118, 139, 136, 132, 132, 139, 133, 132, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443842219Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[0]R generation[0..12]: [72, 70, 73, 62, 75, 74, 72, 72, 75, 73, 72, 74]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443856677Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443871266Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[1]R flush_numbers[0..12]: [132, 130, 133, 118, 139, 136, 132, 132, 139, 133, 132, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443885214Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[1]R generation[0..12]: [72, 70, 73, 62, 75, 74, 72, 72, 75, 73, 72, 74]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443898923Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443913112Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[2]R flush_numbers[0..12]: [132, 130, 133, 118, 139, 136, 132, 132, 139, 133, 132, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.44392693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[2]R generation[0..12]: [72, 70, 73, 62, 75, 74, 72, 72, 75, 73, 72, 74]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443944329Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443958817Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Max found gen is 76","v":0,"Wait for a query_work_queue command to finish before sending IO name":"crucible","level":30,"time":"2024-07-09T21:38:18.443974875Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.443988554Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Next flush: 140","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.444001613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.444014782Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"":"downstairs","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.44402809Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.444041019Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"0","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.444054418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"1","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.444067606Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"client":"2","":"downstairs","session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.444080775Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.444092944Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"41d72f86-0523-4547-9116-7ed945ee74cb is now active with session: 4a2cbfdd-da1b-4baf-9a6f-ad403a41f312","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.444105953Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.444117981Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489,"session_id":"4a2cbfdd-da1b-4baf-9a6f-ad403a41f312"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:18.4441302Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 53 len 40960 data: 7 8 9 10 10 10 10 9 9 8 03/30 Read block 473 len 4096 04/30 Write block 355 len 8192 data: 4 8 05/30 Read block 589 len 20480 06/30 Read block 149 len 36864 07/30 Flush 08/30 Write block 187 len 12288 data: 5 4 6 09/30 Flush 10/30 Write block 400 len 8192 data: 4 3 11/30 Read block 491 len 32768 12/30 Read block 474 len 32768 13/30 Read block 208 len 40960 14/30 Write block 446 len 16384 data: 8 8 7 7 15/30 Read block 189 len 36864 16/30 Read block 132 len 28672 17/30 Read block 146 len 32768 18/30 Write block 467 len 12288 data: 8 8 8 19/30 Write block 330 len 24576 data: 6 8 8 6 5 5 20/30 Read block 571 len 24576 21/30 Write block 63 len 24576 data: 6 4 4 5 5 5 22/30 Read block 189 len 16384 23/30 Read block 436 len 8192 24/30 Write block 7 len 40960 data: 3 4 5 5 5 7 8 10 9 10 25/30 Write block 447 len 40960 data: 9 8 8 7 9 7 6 7 7 7 26/30 Write block 323 len 28672 data: 7 7 9 8 8 8 6 27/30 Write block 262 len 24576 data: 6 6 6 7 7 6 28/30 Write block 222 len 16384 data: 8 7 7 6 29/30 Write block 50 len 8192 data: 3 4 30/30 Write block 479 len 24576 data: 4 4 4 4 6 6 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Read 8 Done Done Done false 19 Acked 1018 Write 3 Done Done Done false 20 Acked 1019 Write 6 Done Done Done false 21 Acked 1020 Read 6 Done Done Done false 22 Acked 1021 Write 6 Done Done Done false 23 Acked 1022 Read 4 Done Done Done false 24 Acked 1023 Read 2 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Write 10 Done Done Done false 27 Acked 1026 Write 7 Done Done Done false 28 Acked 1027 Write 6 Done Done Done false 29 Acked 1028 Write 4 Done Done Done false 30 Acked 1029 Write 2 Done Done Done false 31 Acked 1030 Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 13 13 13 39 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:18.471208146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1489} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 132 132 132 T T T 1 020-039 70 70 70 130 130 130 F F F 2 040-059 76 76 76 141 141 141 T T T 3 060-079 76 76 76 141 141 141 T T T 4 080-099 75 75 75 139 139 139 F F F 5 100-119 74 74 74 136 136 136 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 72 72 72 132 132 132 F F F 8 160-179 75 75 75 139 139 139 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 72 72 72 132 132 132 F F F 11 220-239 74 74 74 137 137 137 T T T 12 240-259 71 71 71 131 131 131 F F F 13 260-279 74 74 74 137 137 137 T T T 14 280-299 74 74 74 137 137 137 F F F 15 300-319 75 75 75 139 139 139 F F F 16 320-339 74 74 74 138 138 138 T T T 17 340-359 76 76 76 141 141 141 F F F 18 360-379 73 73 73 133 133 133 F F F 19 380-399 74 74 74 137 137 137 F F F 20 400-419 76 76 76 143 143 143 F F F 21 420-439 74 74 74 135 135 135 F F F 22 440-459 76 76 76 143 143 143 T T T 23 460-479 75 75 75 139 139 139 T T T 24 480-499 75 75 75 139 139 139 T T T 25 500-519 72 72 72 132 132 132 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 75 75 75 139 139 139 F F F 28 560-579 75 75 75 139 139 139 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 76, Max flush: 143 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 77 77 77 144 144 144 F F F 1 020-039 70 70 70 130 130 130 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 77 77 77 144 144 144 F F F 4 080-099 75 75 75 139 139 139 F F F 5 100-119 74 74 74 136 136 136 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 72 72 72 132 132 132 F F F 8 160-179 75 75 75 139 139 139 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 72 72 72 132 132 132 F F F 11 220-239 77 77 77 144 144 144 F F F 12 240-259 71 71 71 131 131 131 F F F 13 260-279 77 77 77 144 144 144 F F F 14 280-299 74 74 74 137 137 137 F F F 15 300-319 75 75 75 139 139 139 F F F 16 320-339 77 77 77 144 144 144 F F F 17 340-359 76 76 76 141 141 141 F F F 18 360-379 73 73 73 133 133 133 F F F 19 380-399 74 74 74 137 137 137 F F F 20 400-419 76 76 76 143 143 143 F F F 21 420-439 74 74 74 135 135 135 F F F 22 440-459 77 77 77 144 144 144 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 77 77 77 144 144 144 F F F 25 500-519 72 72 72 132 132 132 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 75 75 75 139 139 139 F F F 28 560-579 75 75 75 139 139 139 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 77, Max flush: 144 Begin loop 39 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.761668032Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.76238409Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.762413107Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497} {"msg":"Crucible e4c82bb5-3ee9-495b-a382-c5d19a9f327e has session id: 5b2edc74-55f5-441a-9b76-2c32041ad190","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.762435745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Upstairs opts: Upstairs UUID: e4c82bb5-3ee9-495b-a382-c5d19a9f327e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.762468582Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Crucible stats registered with UUID: e4c82bb5-3ee9-495b-a382-c5d19a9f327e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.762494659Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.762815948Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497} {"msg":"e4c82bb5-3ee9-495b-a382-c5d19a9f327e active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.762886071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.762915398Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.762942555Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.762968372Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"1","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.76299166Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"1","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.763013958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"2","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.763038176Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"2","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.763061753Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"io task","client":"2","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.763086091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"io task","client":"1","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.763111028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"io task","client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"ds_connection connected from Ok(127.0.0.1:62093)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.763255664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"io task","client":"1","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"ds_connection connected from Ok(127.0.0.1:36884)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.763290561Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"io task","client":"2","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:32.763322417Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"io task","client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:32.763466083Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"io task","client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:32.76349517Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.763513398Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.763537296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.763562384Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"io task","client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.764120398Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"1","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.764147276Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"1","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.764162654Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"2","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.764176333Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"2","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.764762605Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"1","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.764789832Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"2","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.765277904Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"1","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.765304171Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"1","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.76531899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.765331408Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"2","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.765348547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"2","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:32.765362095Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.764233233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"io task","client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"ds_connection connected from Ok(127.0.0.1:46358)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.764436123Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"io task","client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.765546283Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.765571641Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766133735Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766664442Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.76668667Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[0]R flush_numbers[0..12]: [144, 130, 144, 144, 139, 136, 132, 132, 139, 142, 132, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766701569Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[0]R generation[0..12]: [77, 70, 77, 77, 75, 74, 72, 72, 75, 76, 72, 77]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766715987Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766730286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[1]R flush_numbers[0..12]: [144, 130, 144, 144, 139, 136, 132, 132, 139, 142, 132, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766744344Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[1]R generation[0..12]: [77, 70, 77, 77, 75, 74, 72, 72, 75, 76, 72, 77]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766758133Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766772451Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[2]R flush_numbers[0..12]: [144, 130, 144, 144, 139, 136, 132, 132, 139, 142, 132, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.76679022Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[2]R generation[0..12]: [77, 70, 77, 77, 75, 74, 72, 72, 75, 76, 72, 77]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766804468Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766818817Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"Wait for a query_work_queue command to finish before sending IO msg":"Max found gen is 78","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766835125Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766848524Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Next flush: 145","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766864692Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766881121Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"":"downstairs","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766897849Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766914417Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"0","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766942045Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"1","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.766962372Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"client":"2","":"downstairs","session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.76698848Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.767016847Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"e4c82bb5-3ee9-495b-a382-c5d19a9f327e is now active with session: 5b2edc74-55f5-441a-9b76-2c32041ad190","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.767046514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.767079071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497,"session_id":"5b2edc74-55f5-441a-9b76-2c32041ad190"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:42.767107058Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 108 len 40960 data: 8 8 7 8 9 9 10 9 7 7 02/30 Write block 263 len 4096 data: 7 03/30 Read block 507 len 16384 04/30 Write block 160 len 40960 data: 7 5 4 4 4 5 6 5 3 4 05/30 Read block 219 len 36864 06/30 Read block 291 len 28672 07/30 Write block 556 len 24576 data: 8 9 10 8 8 9 08/30 Read block 255 len 16384 09/30 Read block 377 len 4096 10/30 Write block 300 len 12288 data: 9 8 6 11/30 Read block 598 len 8192 12/30 Read block 297 len 12288 13/30 Read block 386 len 12288 14/30 Write block 20 len 16384 data: 5 5 2 4 15/30 Write block 84 len 16384 data: 6 5 7 7 16/30 Read block 517 len 4096 17/30 Write block 340 len 28672 data: 4 6 7 7 7 7 7 18/30 Read block 29 len 32768 19/30 Write block 351 len 40960 data: 4 4 6 4 5 9 9 10 9 8 20/30 Write block 208 len 32768 data: 12 10 9 10 9 9 7 8 21/30 Read block 576 len 40960 22/30 Read block 481 len 32768 23/30 Read block 317 len 36864 24/30 Write block 21 len 8192 data: 6 3 25/30 Read block 555 len 24576 26/30 Flush 27/30 Write block 485 len 36864 data: 7 8 9 9 9 11 10 4 5 28/30 Read block 79 len 8192 29/30 Read block 505 len 16384 30/30 Write block 77 len 32768 data: 4 5 4 4 4 5 6 7 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Read 2 Done Done Done false 30 Acked 1029 Read 4 Done Done Done false 31 Acked 1030 Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:42.794440368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1497} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 77 77 77 144 144 144 F F F 1 020-039 78 78 78 146 146 146 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 77 77 77 144 144 144 T T T 4 080-099 78 78 78 145 145 145 T T T 5 100-119 78 78 78 145 145 145 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 72 72 72 132 132 132 F F F 8 160-179 78 78 78 145 145 145 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 78 78 78 145 145 145 F F F 11 220-239 77 77 77 144 144 144 F F F 12 240-259 71 71 71 131 131 131 F F F 13 260-279 78 78 78 145 145 145 F F F 14 280-299 74 74 74 137 137 137 F F F 15 300-319 78 78 78 145 145 145 F F F 16 320-339 77 77 77 144 144 144 F F F 17 340-359 78 78 78 145 145 145 F F F 18 360-379 78 78 78 145 145 145 F F F 19 380-399 74 74 74 137 137 137 F F F 20 400-419 76 76 76 143 143 143 F F F 21 420-439 74 74 74 135 135 135 F F F 22 440-459 77 77 77 144 144 144 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 77 77 77 144 144 144 T T T 25 500-519 72 72 72 132 132 132 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 78 78 78 145 145 145 F F F 28 560-579 78 78 78 145 145 145 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 78, Max flush: 146 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 77 77 77 144 144 144 F F F 1 020-039 78 78 78 146 146 146 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 79 79 79 147 147 147 F F F 4 080-099 79 79 79 147 147 147 F F F 5 100-119 78 78 78 145 145 145 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 72 72 72 132 132 132 F F F 8 160-179 78 78 78 145 145 145 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 78 78 78 145 145 145 F F F 11 220-239 77 77 77 144 144 144 F F F 12 240-259 71 71 71 131 131 131 F F F 13 260-279 78 78 78 145 145 145 F F F 14 280-299 74 74 74 137 137 137 F F F 15 300-319 78 78 78 145 145 145 F F F 16 320-339 77 77 77 144 144 144 F F F 17 340-359 78 78 78 145 145 145 F F F 18 360-379 78 78 78 145 145 145 F F F 19 380-399 74 74 74 137 137 137 F F F 20 400-419 76 76 76 143 143 143 F F F 21 420-439 74 74 74 135 135 135 F F F 22 440-459 77 77 77 144 144 144 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 79 79 79 147 147 147 F F F 25 500-519 72 72 72 132 132 132 F F F 26 520-539 63 63 63 119 119 119 F F F 27 540-559 78 78 78 145 145 145 F F F 28 560-579 78 78 78 145 145 145 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 79, Max flush: 147 Begin loop 40 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.995920843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.996217454Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.996237852Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505} {"msg":"Crucible fc1a6921-ca7e-4697-a4fb-3980d9a04de0 has session id: 83aeb48f-b168-4ec1-8100-57b5c43145a9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.996776559Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Upstairs opts: Upstairs UUID: fc1a6921-ca7e-4697-a4fb-3980d9a04de0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.996818504Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Crucible stats registered with UUID: fc1a6921-ca7e-4697-a4fb-3980d9a04de0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.996845392Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.99716804Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505} {"msg":"fc1a6921-ca7e-4697-a4fb-3980d9a04de0 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997250972Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.99727053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"0","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997285448Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"0","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997299187Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997312645Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997328984Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"2","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997353161Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"2","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.99736994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"io task","client":"0","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997384098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"io task","client":"2","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997398367Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"io task","client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:56.997597907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"io task","client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:56.997639893Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"io task","client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:38:56.997726874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:65428)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997752412Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"io task","client":"0","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.99776802Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997781429Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997898057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"io task","client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:41571)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.997935344Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"io task","client":"2","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.998360642Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"0","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.998387059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"0","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.998485359Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"2","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.998555442Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"2","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.998830065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"0","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.998936685Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"2","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.999345914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"0","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.99938449Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"0","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.999412617Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.999439155Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"2","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.999471492Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"2","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:38:56.999490179Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:06.998372796Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"io task","client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:41772)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:06.998556177Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"io task","client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:06.999564907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:06.999595904Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000185496Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000769828Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000792526Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[0]R flush_numbers[0..12]: [144, 146, 144, 147, 147, 145, 132, 132, 145, 142, 145, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000807594Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[0]R generation[0..12]: [77, 78, 77, 79, 79, 78, 72, 72, 78, 76, 78, 77]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000821943Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000836351Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[1]R flush_numbers[0..12]: [144, 146, 144, 147, 147, 145, 132, 132, 145, 142, 145, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.00085065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[1]R generation[0..12]: [77, 78, 77, 79, 79, 78, 72, 72, 78, 76, 78, 77]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000864718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000879167Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[2]R flush_numbers[0..12]: [144, 146, 144, 147, 147, 145, 132, 132, 145, 142, 145, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000898295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[2]R generation[0..12]: [77, 78, 77, 79, 79, 78, 72, 72, 78, 76, 78, 77]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000912474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000926732Z","hostname":Wait for a query_work_queue command to finish before sending IO "w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000943611Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000957119Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Next flush: 148","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000970228Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000983317Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"":"downstairs","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.000997465Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.001018713Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"0","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.001044471Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"1","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.001070368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"client":"2","":"downstairs","session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.001097015Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.001121443Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"fc1a6921-ca7e-4697-a4fb-3980d9a04de0 is now active with session: 83aeb48f-b168-4ec1-8100-57b5c43145a9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.001146781Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.001170978Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505,"session_id":"83aeb48f-b168-4ec1-8100-57b5c43145a9"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:07.001198765Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 94 len 32768 data: 7 7 7 10 9 9 7 7 03/30 Write block 217 len 24576 data: 7 5 5 4 6 9 04/30 Write block 212 len 40960 data: 10 10 8 9 7 8 6 6 5 7 05/30 Write block 552 len 40960 data: 10 7 8 7 9 10 11 9 9 10 06/30 Write block 13 len 4096 data: 9 07/30 Read block 184 len 28672 08/30 Write block 75 len 20480 data: 3 4 5 6 5 09/30 Write block 537 len 24576 data: 5 6 6 6 5 5 10/30 Write block 175 len 4096 data: 6 11/30 Read block 213 len 20480 12/30 Flush 13/30 Flush 14/30 Flush 15/30 Write block 362 len 32768 data: 6 5 5 6 7 6 6 6 16/30 Read block 422 len 16384 17/30 Read block 275 len 12288 18/30 Read block 491 len 12288 19/30 Read block 158 len 24576 20/30 Write block 30 len 40960 data: 10 10 10 9 8 9 9 12 7 7 21/30 Flush 22/30 Write block 29 len 4096 data: 11 23/30 Read block 50 len 28672 24/30 Read block 25 len 8192 25/30 Write block 403 len 32768 data: 4 5 4 6 6 9 10 9 26/30 Write block 505 len 40960 data: 6 7 7 8 6 8 8 8 6 6 27/30 Read block 531 len 12288 28/30 Read block 495 len 24576 29/30 Read block 38 len 12288 30/30 Write block 112 len 32768 data: 10 10 11 10 8 8 5 5 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 1 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Read 2 Done Done Done false 25 Acked 1024 Write 8 Done Done Done false 26 Acked 1025 Write 10 Done Done Done false 27 Acked 1026 Read 3 Done Done Done false 28 Acked 1027 Read 6 Done Done Done false 29 Acked 1028 Read 3 Done Done Done false 30 Acked 1029 Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:39:07.028133996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1505} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 80 80 80 149 149 149 F F F 1 020-039 80 80 80 152 152 152 T T T 2 040-059 77 77 77 144 144 144 F F F 3 060-079 80 80 80 149 149 149 F F F 4 080-099 80 80 80 149 149 149 F F F 5 100-119 80 80 80 149 149 149 T T T 6 120-139 72 72 72 132 132 132 F F F 7 140-159 72 72 72 132 132 132 F F F 8 160-179 80 80 80 149 149 149 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 F F F 12 240-259 71 71 71 131 131 131 F F F 13 260-279 78 78 78 145 145 145 F F F 14 280-299 74 74 74 137 137 137 F F F 15 300-319 78 78 78 145 145 145 F F F 16 320-339 77 77 77 144 144 144 F F F 17 340-359 78 78 78 145 145 145 F F F 18 360-379 80 80 80 152 152 152 F F F 19 380-399 74 74 74 137 137 137 F F F 20 400-419 76 76 76 143 143 143 T T T 21 420-439 74 74 74 135 135 135 F F F 22 440-459 77 77 77 144 144 144 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 79 79 79 147 147 147 F F F 25 500-519 72 72 72 132 132 132 T T T 26 520-539 80 80 80 149 149 149 F F F 27 540-559 80 80 80 149 149 149 F F F 28 560-579 80 80 80 149 149 149 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 80, Max flush: 152 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 80 80 80 149 149 149 F F F 1 020-039 81 81 81 153 153 153 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 80 80 80 149 149 149 F F F 4 080-099 80 80 80 149 149 149 F F F 5 100-119 81 81 81 153 153 153 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 72 72 72 132 132 132 F F F 8 160-179 80 80 80 149 149 149 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 F F F 12 240-259 71 71 71 131 131 131 F F F 13 260-279 78 78 78 145 145 145 F F F 14 280-299 74 74 74 137 137 137 F F F 15 300-319 78 78 78 145 145 145 F F F 16 320-339 77 77 77 144 144 144 F F F 17 340-359 78 78 78 145 145 145 F F F 18 360-379 80 80 80 152 152 152 F F F 19 380-399 74 74 74 137 137 137 F F F 20 400-419 81 81 81 153 153 153 F F F 21 420-439 74 74 74 135 135 135 F F F 22 440-459 77 77 77 144 144 144 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 79 79 79 147 147 147 F F F 25 500-519 81 81 81 153 153 153 F F F 26 520-539 80 80 80 149 149 149 F F F 27 540-559 80 80 80 149 149 149 F F F 28 560-579 80 80 80 149 149 149 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 81, Max flush: 153 Begin loop 41 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.242893025Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.243228682Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.2432476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513} {"msg":"Crucible a8f0ef5f-45ce-461d-b783-13aa1c975040 has session id: a8847709-db98-4a12-94fd-0c3c6b495d4d","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.243591586Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Upstairs opts: Upstairs UUID: a8f0ef5f-45ce-461d-b783-13aa1c975040, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.243640131Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Crucible stats registered with UUID: a8f0ef5f-45ce-461d-b783-13aa1c975040","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.243671547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.243923533Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513} {"msg":"a8f0ef5f-45ce-461d-b783-13aa1c975040 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.243964689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.243987906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244011164Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244033052Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"1","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.24405476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"1","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244075927Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"2","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244097535Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"2","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244118033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"io task","client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244139801Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"io task","client":"1","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244178017Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"io task","client":"2","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:39:21.24435019Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"io task","client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:39:21.244412174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"io task","client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:39:21.24445189Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"ds_connection connected from Ok(127.0.0.1:43117)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244475958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"io task","client":"1","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244502065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244535392Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"ds_connection connected from Ok(127.0.0.1:45622)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244634912Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"io task","client":"2","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.244716434Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"io task","client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.245330903Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"1","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.24536675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"1","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.245437283Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"2","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.245470659Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"2","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.246022574Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"2","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.246051182Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"1","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.246555382Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"2","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.246582049Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"2","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.246596628Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.246625995Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"1","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.246640933Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"1","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:21.246654252Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.245801092Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"io task","client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"ds_connection connected from Ok(127.0.0.1:65528)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.246173955Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"io task","client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.247214102Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.24724059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.247798674Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248371177Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248393895Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[0]R flush_numbers[0..12]: [149, 153, 144, 149, 149, 153, 132, 132, 149, 142, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248412173Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[0]R generation[0..12]: [80, 81, 77, 80, 80, 81, 72, 72, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248429672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.24844761Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[1]R flush_numbers[0..12]: [149, 153, 144, 149, 149, 153, 132, 132, 149, 142, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248465358Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[1]R generation[0..12]: [80, 81, 77, 80, 80, 81, 72, 72, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248482526Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248504944Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[2]R flush_numbers[0..12]: [149, 153, 144, 149, 149, 153, 132, 132, 149, 142, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248522942Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[2]R generation[0..12]: [80, 81, 77, 80, 80, 81, 72, 72, 80, 76, 80, 80]","v":Wait for a query_work_queue command to finish before sending IO 0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.2485434Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248561539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248577857Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248594495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Next flush: 154","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248610734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248627052Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"":"downstairs","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248643741Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248658849Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"0","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248675547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"1","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248692126Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"client":"2","":"downstairs","session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248708874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248724582Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"a8f0ef5f-45ce-461d-b783-13aa1c975040 is now active with session: a8847709-db98-4a12-94fd-0c3c6b495d4d","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248740901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248758439Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513,"session_id":"a8847709-db98-4a12-94fd-0c3c6b495d4d"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:31.248773918Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 587 len 8192 data: 7 6 02/30 Write block 297 len 4096 data: 7 03/30 Read block 489 len 4096 04/30 Write block 581 len 20480 data: 7 8 6 9 8 05/30 Read block 45 len 12288 06/30 Read block 425 len 28672 07/30 Write block 146 len 20480 data: 6 4 4 4 4 08/30 Read block 103 len 16384 09/30 Read block 180 len 24576 10/30 Read block 70 len 12288 11/30 Write block 534 len 40960 data: 4 7 6 6 7 7 7 6 6 5 12/30 Write block 453 len 24576 data: 7 8 8 8 8 8 13/30 Read block 284 len 24576 14/30 Read block 27 len 20480 15/30 Read block 34 len 24576 16/30 Read block 375 len 32768 17/30 Read block 482 len 12288 18/30 Read block 119 len 16384 19/30 Read block 567 len 36864 20/30 Write block 369 len 32768 data: 7 6 4 3 5 6 7 6 21/30 Flush 22/30 Write block 70 len 8192 data: 6 8 23/30 Write block 587 len 4096 data: 8 24/30 Write block 407 len 8192 data: 7 10 25/30 Read block 530 len 28672 26/30 Write block 403 len 12288 data: 5 6 5 27/30 Write block 240 len 24576 data: 6 4 4 4 4 3 28/30 Write block 374 len 24576 data: 7 8 7 4 4 6 29/30 Read block 562 len 40960 30/30 Write block 384 len 28672 data: 5 8 9 9 11 13 12 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 3 Done Done Done false 28 Acked 1027 Write 6 Done Done Done false 29 Acked 1028 Write 6 Done Done Done false 30 Acked 1029 Read 10 Done Done Done false 31 Acked 1030 Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 26 29 28 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:39:31.27508037Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1513} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 80 80 80 149 149 149 F F F 1 020-039 81 81 81 153 153 153 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 82 82 82 155 155 155 F F F 4 080-099 80 80 80 149 149 149 F F F 5 100-119 81 81 81 153 153 153 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 82 82 82 154 154 154 F F F 8 160-179 80 80 80 149 149 149 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 F F F 12 240-259 71 71 71 131 131 131 T T T 13 260-279 78 78 78 145 145 145 F F F 14 280-299 82 82 82 154 154 154 F F F 15 300-319 78 78 78 145 145 145 F F F 16 320-339 77 77 77 144 144 144 F F F 17 340-359 78 78 78 145 145 145 F F F 18 360-379 82 82 82 154 154 154 T T T 19 380-399 74 74 74 137 137 137 T T T 20 400-419 82 82 82 155 155 155 T T T 21 420-439 74 74 74 135 135 135 F F F 22 440-459 82 82 82 154 154 154 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 79 79 79 147 147 147 F F F 25 500-519 81 81 81 153 153 153 F F F 26 520-539 82 82 82 154 154 154 F F F 27 540-559 82 82 82 154 154 154 F F F 28 560-579 80 80 80 149 149 149 F F F 29 580-599 82 82 82 155 155 155 F F F Max gen: 82, Max flush: 155 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 80 80 80 149 149 149 F F F 1 020-039 81 81 81 153 153 153 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 82 82 82 155 155 155 F F F 4 080-099 80 80 80 149 149 149 F F F 5 100-119 81 81 81 153 153 153 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 82 82 82 154 154 154 F F F 8 160-179 80 80 80 149 149 149 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 F F F 12 240-259 83 83 83 156 156 156 F F F 13 260-279 78 78 78 145 145 145 F F F 14 280-299 82 82 82 154 154 154 F F F 15 300-319 78 78 78 145 145 145 F F F 16 320-339 77 77 77 144 144 144 F F F 17 340-359 78 78 78 145 145 145 F F F 18 360-379 83 83 83 156 156 156 F F F 19 380-399 83 83 83 156 156 156 F F F 20 400-419 83 83 83 156 156 156 F F F 21 420-439 74 74 74 135 135 135 F F F 22 440-459 82 82 82 154 154 154 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 79 79 79 147 147 147 F F F 25 500-519 81 81 81 153 153 153 F F F 26 520-539 82 82 82 154 154 154 F F F 27 540-559 82 82 82 154 154 154 F F F 28 560-579 80 80 80 149 149 149 F F F 29 580-599 82 82 82 155 155 155 F F F Max gen: 83, Max flush: 156 Begin loop 42 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.490771467Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.49104444Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.491062409Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521} {"msg":"Crucible 3b938e3d-fc9c-44eb-8bb4-c1327aba8468 has session id: d7c3a83e-307b-438c-a698-882983c96d29","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.491423433Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Upstairs opts: Upstairs UUID: 3b938e3d-fc9c-44eb-8bb4-c1327aba8468, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.491466918Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Crucible stats registered with UUID: 3b938e3d-fc9c-44eb-8bb4-c1327aba8468","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.491492096Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.491849321Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521} {"msg":"3b938e3d-fc9c-44eb-8bb4-c1327aba8468 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.491918614Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.491938132Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.49195676Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.491980858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"1","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.492006665Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"1","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.492032753Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"2","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.49205811Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"2","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.492113134Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"io task","client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.492181628Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"io task","client":"1","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.492208615Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"io task","client":"2","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"ds_connection connected from Ok(127.0.0.1:64643)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.492288007Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"io task","client":"1","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:39:45.492429993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"io task","client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:39:45.492473329Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"io task","client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"ds_connection connected from Ok(127.0.0.1:45436)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.492501126Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"io task","client":"2","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:39:45.492530393Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.492554171Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.492577639Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.492654671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"io task","client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.493137463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"1","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.49316523Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"1","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.493181019Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"2","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.493195027Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"2","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.493739123Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"2","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.493860591Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"1","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.494200268Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"2","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.494237894Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"2","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.494258012Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.494398678Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"1","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.494423705Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"1","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:45.494438594Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.493149718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"io task","client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"ds_connection connected from Ok(127.0.0.1:54544)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.493349548Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"io task","client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.494400634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.494428601Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495050519Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495611664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495634521Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[0]R flush_numbers[0..12]: [149, 153, 144, 155, 149, 153, 132, 154, 149, 142, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.49564989Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[0]R generation[0..12]: [80, 81, 77, 82, 80, 81, 72, 82, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495664998Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495679657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[1]R flush_numbers[0..12]: [149, 153, 144, 155, 149, 153, 132, 154, 149, 142, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495693626Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[1]R generation[0..12]: [80, 81, 77, 82, 80, 81, 72, 82, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495707374Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495725112Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[2]R flush_numbers[0..12]: [149, 153, 144, 155, 149, 153, 132, 154, 149, 142, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495739331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[2]R generation[0..12]: [80, 81, 77, 82, 80, 81, 72, 82, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.49575308Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495767308Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Max found gen is 84","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495783957Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Generation requested: 84 >= found:84","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495797355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Next flush: 157","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495810294Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495823173Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"":"downstairs","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495836701Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.4958486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"0","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495862059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"1","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495875188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"client":"2","":"downstairs","session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495888926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495901295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"3b938e3d-fc9c-44eb-8bb4-c1327aba8468 is now active with session: d7c3a83e-307b-438c-a698-882983c96d29","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495914054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495928082Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521,"session_id":"d7c3a83e-307b-438c-a698-882983c96d29"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:39:55.495940511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 86 len 40960 data: 8 8 8 7 7 7 7 7 8 8 02/30 Read block 46 len 12288 03/30 Read block 30 len 20480 04/30 Read block 279 len 12288 05/30 Read block 196 len 8192 06/30 Flush 07/30 Write block 255 len 20480 data: 5 5 4 4 5 08/30 Read block 343 len 4096 09/30 Write block 562 len 12288 data: 10 10 10 10/30 Read block 412 len 20480 11/30 Read block 261 len 36864 12/30 Write block 512 len 36864 data: 9 7 7 6 5 5 4 4 4 13/30 Read block 37 len 12288 14/30 Flush 15/30 Write block 271 len 12288 data: 5 4 6 16/30 Read block 134 len 40960 17/30 Read block 207 len 4096 18/30 Read block 520 len 28672 19/30 Flush 20/30 Write block 80 len 32768 data: 5 5 6 7 8 6 9 9 21/30 Read block 353 len 12288 22/30 Flush 23/30 Write block 299 len 12288 data: 8 10 9 24/30 Read block 464 len 8192 25/30 Write block 35 len 20480 data: 10 10 13 8 8 26/30 Write block 333 len 36864 data: 7 6 6 4 5 4 5 5 7 27/30 Read block 136 len 20480 28/30 Write block 88 len 8192 data: 9 8 29/30 Read block 86 len 32768 30/30 Write block 553 len 16384 data: 8 9 8 10 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 3 Done Done Done false 24 Acked 1023 Read 2 Done Done Done false 25 Acked 1024 Write 5 Done Done Done false 26 Acked 1025 Write 9 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 8 Done Done Done false 30 Acked 1029 Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:39:55.525188212Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1521} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 80 80 80 149 149 149 F F F 1 020-039 81 81 81 153 153 153 T T T 2 040-059 77 77 77 144 144 144 F F F 3 060-079 82 82 82 155 155 155 F F F 4 080-099 84 84 84 160 160 160 T T T 5 100-119 81 81 81 153 153 153 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 82 82 82 154 154 154 F F F 8 160-179 80 80 80 149 149 149 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 F F F 12 240-259 84 84 84 158 158 158 F F F 13 260-279 84 84 84 159 159 159 F F F 14 280-299 82 82 82 154 154 154 T T T 15 300-319 78 78 78 145 145 145 T T T 16 320-339 77 77 77 144 144 144 T T T 17 340-359 78 78 78 145 145 145 T T T 18 360-379 83 83 83 156 156 156 F F F 19 380-399 83 83 83 156 156 156 F F F 20 400-419 83 83 83 156 156 156 F F F 21 420-439 74 74 74 135 135 135 F F F 22 440-459 82 82 82 154 154 154 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 79 79 79 147 147 147 F F F 25 500-519 84 84 84 158 158 158 F F F 26 520-539 84 84 84 158 158 158 F F F 27 540-559 82 82 82 154 154 154 T T T 28 560-579 84 84 84 158 158 158 F F F 29 580-599 82 82 82 155 155 155 F F F Max gen: 84, Max flush: 160 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 80 80 80 149 149 149 F F F 1 020-039 85 85 85 161 161 161 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 82 82 82 155 155 155 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 81 81 81 153 153 153 F F F 6 120-139 72 72 72 132 132 132 F F F 7 140-159 82 82 82 154 154 154 F F F 8 160-179 80 80 80 149 149 149 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 F F F 12 240-259 84 84 84 158 158 158 F F F 13 260-279 84 84 84 159 159 159 F F F 14 280-299 85 85 85 161 161 161 F F F 15 300-319 85 85 85 161 161 161 F F F 16 320-339 85 85 85 161 161 161 F F F 17 340-359 85 85 85 161 161 161 F F F 18 360-379 83 83 83 156 156 156 F F F 19 380-399 83 83 83 156 156 156 F F F 20 400-419 83 83 83 156 156 156 F F F 21 420-439 74 74 74 135 135 135 F F F 22 440-459 82 82 82 154 154 154 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 79 79 79 147 147 147 F F F 25 500-519 84 84 84 158 158 158 F F F 26 520-539 84 84 84 158 158 158 F F F 27 540-559 85 85 85 161 161 161 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 82 82 82 155 155 155 F F F Max gen: 85, Max flush: 161 Begin loop 43 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.787240984Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.787590359Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.787610517Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529} {"msg":"Crucible 42dc8530-964d-4d83-a95e-561882c12b83 has session id: 1eaf751f-d67c-45da-9617-76caa40c4e51","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.787993009Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Upstairs opts: Upstairs UUID: 42dc8530-964d-4d83-a95e-561882c12b83, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788032395Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Crucible stats registered with UUID: 42dc8530-964d-4d83-a95e-561882c12b83","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788048304Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788429346Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529} {"msg":"42dc8530-964d-4d83-a95e-561882c12b83 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788455133Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788470562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"0","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.78848589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"0","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788500059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788513887Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788529086Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"2","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788552714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"2","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788577241Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"io task","client":"0","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788601669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"io task","client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788618367Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"io task","client":"2","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"ds_connection connected from Ok(127.0.0.1:51133)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788809348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"io task","client":"0","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"ds_connection connected from Ok(127.0.0.1:43850)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.788865892Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"io task","client":"2","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:09.78909585Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"io task","client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:09.789137825Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"io task","client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:09.789169112Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.78919505Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.789219457Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.789254294Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"io task","client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.789618508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"2","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.789651195Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"2","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.789678972Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"0","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.789706169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"0","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.790228957Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"0","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.790265034Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"2","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.790755105Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"0","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.790781522Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"0","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.790796621Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.790828428Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"2","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.790844046Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"2","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:09.790857505Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.791048271Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"io task","client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"ds_connection connected from Ok(127.0.0.1:39195)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.791276449Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"io task","client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.7922687Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.792294958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.792791318Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793345503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793373581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[0]R flush_numbers[0..12]: [149, 161, 144, 155, 161, 153, 132, 154, 149, 142, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793392409Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[0]R generation[0..12]: [80, 85, 77, 82, 85, 81, 72, 82, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793410377Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793427915Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[1]R flush_numbers[0..12]: [149, 161, 144, 155, 161, 153, 132, 154, 149, 142, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793445584Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[1]R generation[0..12]: [80, 85, 77, 82, 85, 81, 72, 82, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793462852Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.79348633Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[2]R flush_numbers[0..12]: [149, 161, 144, 155, 161, 153, 132, 154, 149, 142, 149, 149]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:40:19.793504808Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[2]R generation[0..12]: [80, 85, 77, 82, 85, 81, 72, 82, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793525106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793542874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Max found gen is 86","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793559172Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Generation requested: 86 >= found:86","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793575751Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Next flush: 162","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793591939Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793608437Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"":"downstairs","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793625326Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793640514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"0","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793657653Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"1","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793674461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"client":"2","":"downstairs","session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793691579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793706858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"42dc8530-964d-4d83-a95e-561882c12b83 is now active with session: 1eaf751f-d67c-45da-9617-76caa40c4e51","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793722956Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793740044Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529,"session_id":"1eaf751f-d67c-45da-9617-76caa40c4e51"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:19.793755613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 129 len 8192 data: 4 4 02/30 Write block 588 len 28672 data: 7 8 8 8 7 8 11 03/30 Read block 49 len 20480 04/30 Read block 47 len 8192 05/30 Write block 100 len 16384 data: 8 8 8 9 06/30 Write block 74 len 12288 data: 6 4 5 07/30 Read block 424 len 20480 08/30 Read block 152 len 40960 09/30 Write block 144 len 16384 data: 6 5 7 5 10/30 Read block 56 len 40960 11/30 Write block 428 len 40960 data: 10 10 10 7 7 8 8 7 5 5 12/30 Write block 379 len 4096 data: 7 13/30 Write block 70 len 8192 data: 7 9 14/30 Read block 312 len 32768 15/30 Write block 12 len 24576 data: 8 10 11 10 11 10 16/30 Read block 400 len 12288 17/30 Write block 262 len 24576 data: 7 8 7 8 8 7 18/30 Write block 368 len 24576 data: 7 8 7 5 4 6 19/30 Write block 536 len 12288 data: 7 7 8 20/30 Read block 474 len 12288 21/30 Write block 394 len 40960 data: 12 11 8 6 3 5 5 4 3 6 22/30 Read block 243 len 4096 23/30 Write block 394 len 24576 data: 13 12 9 7 4 6 24/30 Read block 231 len 40960 25/30 Write block 491 len 4096 data: 11 26/30 Write block 435 len 20480 data: 8 6 6 8 6 27/30 Write block 517 len 8192 data: 6 5 28/30 Read block 86 len 32768 29/30 Read block 315 len 40960 30/30 Read block 491 len 20480 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 5 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 8 Done Done Done false 30 Acked 1029 Read 10 Done Done Done false 31 Acked 1030 Read 5 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 26 29 28 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:19.819074924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1529} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 86 86 86 162 162 162 F F F 1 020-039 85 85 85 161 161 161 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 86 86 86 162 162 162 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 86 86 86 162 162 162 F F F 6 120-139 86 86 86 162 162 162 F F F 7 140-159 86 86 86 162 162 162 F F F 8 160-179 80 80 80 149 149 149 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 F F F 12 240-259 84 84 84 158 158 158 F F F 13 260-279 86 86 86 162 162 162 F F F 14 280-299 85 85 85 161 161 161 F F F 15 300-319 85 85 85 161 161 161 F F F 16 320-339 85 85 85 161 161 161 F F F 17 340-359 85 85 85 161 161 161 F F F 18 360-379 86 86 86 162 162 162 F F F 19 380-399 86 86 86 162 162 162 F F F 20 400-419 86 86 86 162 162 162 F F F 21 420-439 86 86 86 162 162 162 T T T 22 440-459 82 82 82 154 154 154 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 86 86 86 162 162 162 F F F 25 500-519 84 84 84 158 158 158 T T T 26 520-539 86 86 86 162 162 162 F F F 27 540-559 85 85 85 161 161 161 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 86 86 86 162 162 162 F F F Max gen: 86, Max flush: 162 A Difference in extent metadata is expected here On loop 43 Verifying data now No change after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 86 86 86 162 162 162 F F F 1 020-039 85 85 85 161 161 161 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 86 86 86 162 162 162 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 86 86 86 162 162 162 F F F 6 120-139 86 86 86 162 162 162 F F F 7 140-159 86 86 86 162 162 162 F F F 8 160-179 80 80 80 149 149 149 F F F 9 180-199 76 76 76 142 142 142 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 F F F 12 240-259 84 84 84 158 158 158 F F F 13 260-279 86 86 86 162 162 162 F F F 14 280-299 85 85 85 161 161 161 F F F 15 300-319 85 85 85 161 161 161 F F F 16 320-339 85 85 85 161 161 161 F F F 17 340-359 85 85 85 161 161 161 F F F 18 360-379 86 86 86 162 162 162 F F F 19 380-399 86 86 86 162 162 162 F F F 20 400-419 86 86 86 162 162 162 F F F 21 420-439 87 87 87 163 163 163 F F F 22 440-459 82 82 82 154 154 154 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 86 86 86 162 162 162 F F F 25 500-519 87 87 87 163 163 163 F F F 26 520-539 86 86 86 162 162 162 F F F 27 540-559 85 85 85 161 161 161 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 86 86 86 162 162 162 F F F Max gen: 87, Max flush: 163 Begin loop 44 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.012091986Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.012697736Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.012727133Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537} {"msg":"Crucible cccee582-c8e3-4afb-8055-916e4e5e0b0c has session id: 4b5f4625-82e6-48a5-bc8f-ed60164c267b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.012766449Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Upstairs opts: Upstairs UUID: cccee582-c8e3-4afb-8055-916e4e5e0b0c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.012803875Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Crucible stats registered with UUID: cccee582-c8e3-4afb-8055-916e4e5e0b0c","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.012827133Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013149461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537} {"msg":"cccee582-c8e3-4afb-8055-916e4e5e0b0c active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013221084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013246491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"0","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.01326295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"0","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013278208Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013292117Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013305785Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"2","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013319624Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"2","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013343072Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"io task","client":"0","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.01335754Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"io task","client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013371609Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"io task","client":"2","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"ds_connection connected from Ok(127.0.0.1:59472)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013530523Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"io task","client":"0","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:34.013570579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"io task","client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:34.013603276Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"io task","client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:34.013629423Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013709405Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013725764Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"ds_connection connected from Ok(127.0.0.1:43527)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013740112Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"io task","client":"2","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.013842472Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"io task","client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.014360731Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"2","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.014387888Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"2","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.014402937Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"0","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.014416625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"0","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.014998798Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"2","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.015152342Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"0","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.015488589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"2","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.015509487Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"2","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.015527235Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.015606517Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"0","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.015625516Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"0","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:34.015639284Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.015680456Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"io task","client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"ds_connection connected from Ok(127.0.0.1:37705)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.016064728Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"io task","client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.01695486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.016986806Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.017518564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018141292Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018167969Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[0]R flush_numbers[0..12]: [162, 161, 144, 162, 161, 162, 162, 162, 149, 142, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018186278Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[0]R generation[0..12]: [86, 85, 77, 86, 85, 86, 86, 86, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018204126Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018222254Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[1]R flush_numbers[0..12]: [162, 161, 144, 162, 161, 162, 162, 162, 149, 142, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018240052Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[1]R generation[0..12]: [86, 85, 77, 86, 85, 86, 86, 86, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.01826146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018279768Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[2]R flush_numbers[0..12]: [162, 161, 144, 162, 161, 162, 162, 162, 149, 142, 149, 149]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:40:44.018297127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[2]R generation[0..12]: [86, 85, 77, 86, 85, 86, 86, 86, 80, 76, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018316925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018335073Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Max found gen is 88","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.0183657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018383058Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Next flush: 164","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018399346Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018415805Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"":"downstairs","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018432653Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018447652Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"0","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.01846423Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"1","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018480918Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"client":"2","":"downstairs","session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018497687Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2024-07-09T21:40:44.018513175Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"cccee582-c8e3-4afb-8055-916e4e5e0b0c is now active with session: 4b5f4625-82e6-48a5-bc8f-ed60164c267b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018532863Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.018548082Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537,"session_id":"4b5f4625-82e6-48a5-bc8f-ed60164c267b"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:44.01856362Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 372 len 24576 data: 5 7 8 9 8 5 02/30 Write block 449 len 12288 data: 9 8 10 03/30 Write block 540 len 16384 data: 8 7 7 6 04/30 Write block 431 len 16384 data: 8 8 9 9 05/30 Write block 285 len 20480 data: 7 5 6 9 10 06/30 Read block 293 len 32768 07/30 Read block 506 len 8192 08/30 Read block 12 len 28672 09/30 Write block 60 len 28672 data: 10 10 9 7 5 5 6 10/30 Write block 318 len 28672 data: 7 8 9 9 7 8 8 11/30 Flush 12/30 Write block 531 len 12288 data: 3 3 4 13/30 Write block 191 len 4096 data: 11 14/30 Write block 350 len 36864 data: 5 5 5 7 5 6 10 10 11 15/30 Write block 100 len 4096 data: 9 16/30 Read block 422 len 24576 17/30 Flush 18/30 Write block 318 len 20480 data: 8 9 10 10 8 19/30 Read block 518 len 8192 20/30 Read block 469 len 16384 21/30 Write block 541 len 40960 data: 8 8 7 5 5 7 7 8 8 10 22/30 Write block 526 len 24576 data: 8 7 7 6 4 4 23/30 Write block 157 len 36864 data: 5 6 6 8 6 5 5 5 6 24/30 Read block 525 len 12288 25/30 Flush 26/30 Flush 27/30 Write block 432 len 28672 data: 9 10 10 9 7 7 9 28/30 Write block 302 len 36864 data: 7 6 7 9 9 7 7 8 7 29/30 Read block 77 len 12288 30/30 Read block 573 len 20480 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 7 Done Done Done false 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Read 3 Done Done Done false 30 Acked 1029 Read 5 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:44.048284735Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1537} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 86 86 86 162 162 162 F F F 1 020-039 85 85 85 161 161 161 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 88 88 88 164 164 164 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 88 88 88 165 165 165 F F F 6 120-139 86 86 86 162 162 162 F F F 7 140-159 88 88 88 166 166 166 F F F 8 160-179 88 88 88 166 166 166 F F F 9 180-199 88 88 88 165 165 165 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 F F F 12 240-259 84 84 84 158 158 158 F F F 13 260-279 86 86 86 162 162 162 F F F 14 280-299 88 88 88 164 164 164 F F F 15 300-319 88 88 88 166 166 166 T T T 16 320-339 88 88 88 166 166 166 F F F 17 340-359 88 88 88 165 165 165 F F F 18 360-379 88 88 88 164 164 164 F F F 19 380-399 86 86 86 162 162 162 F F F 20 400-419 86 86 86 162 162 162 F F F 21 420-439 88 88 88 164 164 164 T T T 22 440-459 88 88 88 164 164 164 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 86 86 86 162 162 162 F F F 25 500-519 87 87 87 163 163 163 F F F 26 520-539 88 88 88 166 166 166 F F F 27 540-559 88 88 88 166 166 166 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 86 86 86 162 162 162 F F F Max gen: 88, Max flush: 166 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 86 86 86 162 162 162 F F F 1 020-039 85 85 85 161 161 161 F F F 2 040-059 77 77 77 144 144 144 F F F 3 060-079 88 88 88 164 164 164 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 88 88 88 165 165 165 F F F 6 120-139 86 86 86 162 162 162 F F F 7 140-159 88 88 88 166 166 166 F F F 8 160-179 88 88 88 166 166 166 F F F 9 180-199 88 88 88 165 165 165 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 F F F 12 240-259 84 84 84 158 158 158 F F F 13 260-279 86 86 86 162 162 162 F F F 14 280-299 88 88 88 164 164 164 F F F 15 300-319 89 89 89 167 167 167 F F F 16 320-339 88 88 88 166 166 166 F F F 17 340-359 88 88 88 165 165 165 F F F 18 360-379 88 88 88 164 164 164 F F F 19 380-399 86 86 86 162 162 162 F F F 20 400-419 86 86 86 162 162 162 F F F 21 420-439 89 89 89 167 167 167 F F F 22 440-459 88 88 88 164 164 164 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 86 86 86 162 162 162 F F F 25 500-519 87 87 87 163 163 163 F F F 26 520-539 88 88 88 166 166 166 F F F 27 540-559 88 88 88 166 166 166 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 86 86 86 162 162 162 F F F Max gen: 89, Max flush: 167 Begin loop 45 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.237980625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.238507613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.23853487Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545} {"msg":"Crucible bdcc06bb-7ea7-4dbb-91b5-faf7e8e79b42 has session id: ed9005e1-20c3-4915-baa1-ef35f389a6e9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.238675576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Upstairs opts: Upstairs UUID: bdcc06bb-7ea7-4dbb-91b5-faf7e8e79b42, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.238741419Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Crucible stats registered with UUID: bdcc06bb-7ea7-4dbb-91b5-faf7e8e79b42","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.238771907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239102144Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545} {"msg":"bdcc06bb-7ea7-4dbb-91b5-faf7e8e79b42 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239136Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239151419Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239166207Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239179696Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"1","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239200044Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"1","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239224322Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"2","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239249769Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"2","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239274657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"io task","client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239299084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"io task","client":"1","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239321542Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"io task","client":"2","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:61655)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239495755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"io task","client":"1","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:58.239537451Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"io task","client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:58.239572467Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"io task","client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:41142)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239602614Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"io task","client":"2","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:40:58.239630771Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239652979Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239676797Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.239770228Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"io task","client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.240378147Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"1","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.240414054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"1","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.240440771Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"2","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.240467028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"2","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.241086947Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"2","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.241118414Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"1","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.241619484Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"1","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.241646671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"1","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.241675309Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.241691747Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"2","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.241705836Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"2","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:40:58.241719344Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.240512519Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"io task","client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:53417)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.240570223Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"io task","client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.241664124Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.24171405Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242229878Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242746527Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242772285Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[0]R flush_numbers[0..12]: [162, 161, 144, 164, 161, 165, 162, 166, 166, 165, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242791313Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[0]R generation[0..12]: [86, 85, 77, 88, 85, 88, 86, 88, 88, 88, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242808971Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242826689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[1]R flush_numbers[0..12]: [162, 161, 144, 164, 161, 165, 162, 166, 166, 165, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242845227Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[1]R generation[0..12]: [86, 85, 77, 88, 85, 88, 86, 88, 88, 88, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242867535Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242897302Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R flush_numbers[0..12]: [162, 161, 144, 164, 161, 165, 162, 166, 166, 165, 149, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242928239Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[2]R generation[0..12]: [86, 85, 77, 88, 85, 88, 86, 88, 88, 88, 80, 80]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242959716Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.242991513Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243013541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243041708Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Next flush: 168","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243072435Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243103432Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"":"downstairs","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.24312261Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243147757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"0","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243179544Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"1","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243209891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"client":"2","":"downstairs","session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243228609Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243254527Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"bdcc06bb-7ea7-4dbb-91b5-faf7e8e79b42 is now active with session: ed9005e1-20c3-4915-baa1-ef35f389a6e9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243284634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.24332409Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545,"session_id":"ed9005e1-20c3-4915-baa1-ef35f389a6e9"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:08.243342978Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 51 len 32768 data: 5 4 8 9 10 11 11 11 02/30 Read block 581 len 8192 03/30 Read block 283 len 16384 04/30 Read block 207 len 8192 05/30 Read block 231 len 20480 06/30 Write block 452 len 32768 data: 8 8 9 9 9 9 9 7 07/30 Read block 336 len 32768 08/30 Write block 103 len 36864 data: 10 9 8 8 7 9 9 8 9 09/30 Read block 257 len 36864 10/30 Read block 180 len 4096 11/30 Write block 532 len 8192 data: 4 5 12/30 Flush 13/30 Write block 48 len 8192 data: 5 5 14/30 Write block 360 len 24576 data: 9 6 7 6 6 7 15/30 Read block 456 len 28672 16/30 Read block 529 len 8192 17/30 Read block 543 len 16384 18/30 Read block 145 len 36864 19/30 Read block 321 len 32768 20/30 Read block 358 len 36864 21/30 Write block 5 len 4096 data: 3 22/30 Read block 402 len 12288 23/30 Write block 278 len 32768 data: 5 5 6 6 5 6 7 8 24/30 Write block 6 len 24576 data: 2 4 5 6 6 6 25/30 Write block 366 len 28672 data: 8 7 8 9 8 6 6 26/30 Read block 163 len 4096 27/30 Write block 307 len 40960 data: 8 8 9 8 7 8 8 9 6 6 28/30 Write block 322 len 40960 data: 9 9 9 10 9 9 9 7 7 9 29/30 Read block 23 len 8192 30/30 Write block 230 len 36864 data: 6 7 7 7 8 7 7 5 6 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 8 Done Done Done false 25 Acked 1024 Write 6 Done Done Done false 26 Acked 1025 Write 7 Done Done Done false 27 Acked 1026 Read 1 Done Done Done false 28 Acked 1027 Write 10 Done Done Done false 29 Acked 1028 Write 10 Done Done Done false 30 Acked 1029 Read 2 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 23 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:41:08.26865188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1545} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 169 169 169 T T T 1 020-039 85 85 85 161 161 161 F F F 2 040-059 90 90 90 169 169 169 F F F 3 060-079 88 88 88 164 164 164 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 90 90 90 168 168 168 F F F 6 120-139 86 86 86 162 162 162 F F F 7 140-159 88 88 88 166 166 166 F F F 8 160-179 88 88 88 166 166 166 F F F 9 180-199 88 88 88 165 165 165 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 80 80 80 149 149 149 T T T 12 240-259 84 84 84 158 158 158 F F F 13 260-279 86 86 86 162 162 162 T T T 14 280-299 88 88 88 164 164 164 T T T 15 300-319 89 89 89 167 167 167 T T T 16 320-339 88 88 88 166 166 166 T T T 17 340-359 88 88 88 165 165 165 F F F 18 360-379 90 90 90 169 169 169 T T T 19 380-399 86 86 86 162 162 162 F F F 20 400-419 86 86 86 162 162 162 F F F 21 420-439 89 89 89 167 167 167 F F F 22 440-459 90 90 90 168 168 168 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 86 86 86 162 162 162 F F F 25 500-519 87 87 87 163 163 163 F F F 26 520-539 90 90 90 168 168 168 F F F 27 540-559 88 88 88 166 166 166 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 86 86 86 162 162 162 F F F Max gen: 90, Max flush: 169 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 91 91 91 170 170 170 F F F 1 020-039 85 85 85 161 161 161 F F F 2 040-059 90 90 90 169 169 169 F F F 3 060-079 88 88 88 164 164 164 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 90 90 90 168 168 168 F F F 6 120-139 86 86 86 162 162 162 F F F 7 140-159 88 88 88 166 166 166 F F F 8 160-179 88 88 88 166 166 166 F F F 9 180-199 88 88 88 165 165 165 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 91 91 91 170 170 170 F F F 12 240-259 84 84 84 158 158 158 F F F 13 260-279 91 91 91 170 170 170 F F F 14 280-299 91 91 91 170 170 170 F F F 15 300-319 91 91 91 170 170 170 F F F 16 320-339 91 91 91 170 170 170 F F F 17 340-359 88 88 88 165 165 165 F F F 18 360-379 91 91 91 170 170 170 F F F 19 380-399 86 86 86 162 162 162 F F F 20 400-419 86 86 86 162 162 162 F F F 21 420-439 89 89 89 167 167 167 F F F 22 440-459 90 90 90 168 168 168 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 86 86 86 162 162 162 F F F 25 500-519 87 87 87 163 163 163 F F F 26 520-539 90 90 90 168 168 168 F F F 27 540-559 88 88 88 166 166 166 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 86 86 86 162 162 162 F F F Max gen: 91, Max flush: 170 Begin loop 46 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.527321337Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.527670763Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.527703119Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553} {"msg":"Crucible e48075cd-83dc-44b6-8665-48fad06cb71f has session id: e8ab7bc4-87f3-4b08-ba7e-563511df7694","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.52819622Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Upstairs opts: Upstairs UUID: e48075cd-83dc-44b6-8665-48fad06cb71f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528240846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Crucible stats registered with UUID: e48075cd-83dc-44b6-8665-48fad06cb71f","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528261404Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.52850353Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553} {"msg":"e48075cd-83dc-44b6-8665-48fad06cb71f active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528528727Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528544176Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"0","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528559504Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"0","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528573343Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"1","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528587042Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"1","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.5286004Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528614359Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528627908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"io task","client":"0","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528642096Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"io task","client":"1","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528655705Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"io task","client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"ds_connection connected from Ok(127.0.0.1:57255)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.528828658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"io task","client":"1","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"ds_connection connected from Ok(127.0.0.1:41455)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.529062165Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"io task","client":"0","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:41:22.529103561Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"io task","client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:41:22.529131658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"io task","client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:41:22.529149466Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.529162085Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.529175343Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.529291552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"io task","client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.529780613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"0","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.529809791Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"0","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.529825399Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"1","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.529844247Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"1","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.530381304Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"1","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.530507261Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"0","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.530851857Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"1","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.530874745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"1","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.530892043Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.530971885Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"0","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.530991393Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"0","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:22.531004992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.530203216Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"io task","client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"ds_connection connected from Ok(127.0.0.1:53527)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.530411596Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"io task","client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.531586129Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.531620316Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532219747Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532744124Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532778401Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[0]R flush_numbers[0..12]: [170, 161, 169, 164, 161, 168, 162, 166, 166, 165, 149, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532798869Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[0]R generation[0..12]: [91, 85, 90, 88, 85, 90, 86, 88, 88, 88, 80, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532817037Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532835035Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[1]R flush_numbers[0..12]: [170, 161, 169, 164, 161, 168, 162, 166, 166, 165, 149, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532852944Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[1]R generation[0..12]: [91, 85, 90, 88, 85, 90, 86, 88, 88, 88, 80, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532875012Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.53289335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[2]R flush_numbers[0..12]: [170, 161, 169, 164, 161, 168, 162, 166, 166, 165, 149, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532914458Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[2]R generation[0..12]: [91, 85, 90, 88, 85, 90, 86, 88, 88, 88, 80, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532932296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532950054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532968692Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.532998229Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Next flush: 171","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.533028506Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.533046635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"":"downstairs","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.533064693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.53309084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"0","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.533120597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"1","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.533150064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"client":"2","":"downstairs","session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.533179712Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"}Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.533213948Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"e48075cd-83dc-44b6-8665-48fad06cb71f is now active with session: e8ab7bc4-87f3-4b08-ba7e-563511df7694","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.533241595Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.533258034Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553,"session_id":"e8ab7bc4-87f3-4b08-ba7e-563511df7694"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:32.533273492Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 15 len 16384 data: 11 12 11 8 02/30 Read block 27 len 36864 03/30 Write block 494 len 4096 data: 5 04/30 Write block 588 len 16384 data: 8 9 9 9 05/30 Read block 325 len 4096 06/30 Flush 07/30 Write block 448 len 4096 data: 9 08/30 Read block 251 len 4096 09/30 Write block 586 len 24576 data: 8 9 9 10 10 10 10/30 Read block 433 len 4096 11/30 Read block 148 len 20480 12/30 Read block 32 len 20480 13/30 Read block 541 len 16384 14/30 Flush 15/30 Write block 163 len 24576 data: 6 6 7 7 6 4 16/30 Read block 178 len 8192 17/30 Write block 412 len 20480 data: 8 6 6 7 8 18/30 Read block 270 len 4096 19/30 Read block 503 len 8192 20/30 Read block 128 len 24576 21/30 Read block 229 len 8192 22/30 Read block 519 len 32768 23/30 Read block 284 len 24576 24/30 Read block 63 len 28672 25/30 Read block 45 len 32768 26/30 Read block 29 len 20480 27/30 Read block 390 len 24576 28/30 Read block 129 len 40960 29/30 Read block 432 len 4096 30/30 Read block 424 len 12288 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Read 8 Done Done Done false 24 Acked 1023 Read 6 Done Done Done false 25 Acked 1024 Read 7 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Read 6 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Read 1 Done Done Done false 31 Acked 1030 Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 0 1 Done 9 8 9 26 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:41:32.558305991Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1553} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 92 92 92 171 171 171 F F F 1 020-039 85 85 85 161 161 161 F F F 2 040-059 90 90 90 169 169 169 F F F 3 060-079 88 88 88 164 164 164 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 90 90 90 168 168 168 F F F 6 120-139 86 86 86 162 162 162 F F F 7 140-159 88 88 88 166 166 166 F F F 8 160-179 92 92 92 173 173 173 F F F 9 180-199 88 88 88 165 165 165 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 91 91 91 170 170 170 F F F 12 240-259 84 84 84 158 158 158 F F F 13 260-279 91 91 91 170 170 170 F F F 14 280-299 91 91 91 170 170 170 F F F 15 300-319 91 91 91 170 170 170 F F F 16 320-339 91 91 91 170 170 170 F F F 17 340-359 88 88 88 165 165 165 F F F 18 360-379 91 91 91 170 170 170 F F F 19 380-399 86 86 86 162 162 162 F F F 20 400-419 92 92 92 173 173 173 F F F 21 420-439 89 89 89 167 167 167 F F F 22 440-459 92 92 92 172 172 172 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 92 92 92 171 171 171 F F F 25 500-519 87 87 87 163 163 163 F F F 26 520-539 90 90 90 168 168 168 F F F 27 540-559 88 88 88 166 166 166 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 92 92 92 172 172 172 F F F Max gen: 92, Max flush: 173 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 92 92 92 171 171 171 F F F 1 020-039 85 85 85 161 161 161 F F F 2 040-059 90 90 90 169 169 169 F F F 3 060-079 88 88 88 164 164 164 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 90 90 90 168 168 168 F F F 6 120-139 86 86 86 162 162 162 F F F 7 140-159 88 88 88 166 166 166 F F F 8 160-179 92 92 92 173 173 173 F F F 9 180-199 88 88 88 165 165 165 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 91 91 91 170 170 170 F F F 12 240-259 84 84 84 158 158 158 F F F 13 260-279 91 91 91 170 170 170 F F F 14 280-299 91 91 91 170 170 170 F F F 15 300-319 91 91 91 170 170 170 F F F 16 320-339 91 91 91 170 170 170 F F F 17 340-359 88 88 88 165 165 165 F F F 18 360-379 91 91 91 170 170 170 F F F 19 380-399 86 86 86 162 162 162 F F F 20 400-419 92 92 92 173 173 173 F F F 21 420-439 89 89 89 167 167 167 F F F 22 440-459 92 92 92 172 172 172 F F F 23 460-479 77 77 77 144 144 144 F F F 24 480-499 92 92 92 171 171 171 F F F 25 500-519 87 87 87 163 163 163 F F F 26 520-539 90 90 90 168 168 168 F F F 27 540-559 88 88 88 166 166 166 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 92 92 92 172 172 172 F F F Max gen: 92, Max flush: 173 Begin loop 47 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.715284858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.715676499Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.715697147Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561} {"msg":"Crucible 3052d7d9-7437-41b5-8ac9-20b531637b1a has session id: a065d04b-2bb9-4d8f-85eb-babc9ee86b04","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716088639Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Upstairs opts: Upstairs UUID: 3052d7d9-7437-41b5-8ac9-20b531637b1a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716133214Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Crucible stats registered with UUID: 3052d7d9-7437-41b5-8ac9-20b531637b1a","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716157632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716453262Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561} {"msg":"3052d7d9-7437-41b5-8ac9-20b531637b1a active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716550383Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716572051Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"0","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716597778Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"0","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716623306Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716646233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716670441Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"2","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716694739Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"2","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716719816Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"io task","client":"2","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716746913Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"io task","client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716770311Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"io task","client":"0","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:41:42.716814747Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"io task","client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:41:42.716837544Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"io task","client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:41:42.716861122Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.71688478Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716908087Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"ds_connection connected from Ok(127.0.0.1:50805)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.716931495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"io task","client":"2","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.717057712Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"io task","client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"ds_connection connected from Ok(127.0.0.1:51714)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.717100678Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"io task","client":"0","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.717551363Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"2","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.717578071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"2","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.71768715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"0","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.717706458Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"0","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.718071682Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"2","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.718181851Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"0","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.71859482Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"2","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.718617958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"2","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.718636366Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.718659794Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"0","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.718673982Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"0","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:42.718687701Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.718625675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"io task","client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"ds_connection connected from Ok(127.0.0.1:37329)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.718801157Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"io task","client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.719676591Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.719709687Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720203749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720718877Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720737265Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[0]R flush_numbers[0..12]: [171, 161, 169, 164, 161, 168, 162, 166, 173, 165, 149, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720761703Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[0]R generation[0..12]: [92, 85, 90, 88, 85, 90, 86, 88, 92, 88, 80, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720782091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720806959Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[1]R flush_numbers[0..12]: [171, 161, 169, 164, 161, 168, 162, 166, 173, 165, 149, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720830456Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[1]R generation[0..12]: [92, 85, 90, 88, 85, 90, 86, 88, 92, 88, 80, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720850394Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720865163Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[2]R flush_numbers[0..12]: [171, 161, 169, 164, 161, 168, 162, 166, 173, 165, 149, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720879271Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"[2]R generation[0..12]: [92, 85, 90, 88, 85, 90, 86, 88, 92, 88, 80, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.72089301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720911318Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Max found gen is 93","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720932246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Generation requested: 94 >= found:93","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720952754Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Next flush: 174","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720966503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.720979352Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"":"downstairs","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.72099262Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.721004679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"0","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.721018218Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"1","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.721031566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"client":"2","":"downstairs","session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.721045115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.721057444Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"3052d7d9-7437-41b5-8ac9-20b531637b1a is now active with session: a065d04b-2bb9-4d8f-85eb-babc9ee86b04","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.721072672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.721085081Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561,"session_id":"a065d04b-2bb9-4d8f-85eb-babc9ee86b04"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:41:52.72109734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 137 len 16384 data: 6 6 8 7 02/30 Read block 229 len 20480 03/30 Write block 251 len 8192 data: 3 3 04/30 Flush 05/30 Flush 06/30 Write block 445 len 4096 data: 9 07/30 Read block 175 len 12288 08/30 Write block 400 len 32768 data: 6 5 4 7 7 6 7 8 09/30 Read block 362 len 16384 10/30 Read block 85 len 8192 11/30 Write block 51 len 32768 data: 6 5 9 10 11 12 12 12 12/30 Read block 596 len 4096 13/30 Read block 335 len 32768 14/30 Read block 282 len 28672 15/30 Read block 121 len 12288 16/30 Write block 44 len 4096 data: 5 17/30 Write block 369 len 4096 data: 10 18/30 Read block 221 len 20480 19/30 Write block 434 len 40960 data: 11 10 8 8 10 7 9 10 10 9 20/30 Read block 195 len 8192 21/30 Read block 152 len 20480 22/30 Read block 247 len 28672 23/30 Write block 472 len 28672 data: 6 6 5 3 3 4 5 24/30 Write block 357 len 28672 data: 11 12 10 10 7 8 7 25/30 Read block 516 len 28672 26/30 Flush 27/30 Write block 396 len 8192 data: 10 8 28/30 Read block 261 len 36864 29/30 Read block 22 len 28672 30/30 Read block 161 len 24576 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Read 6 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:41:52.748043149Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1561} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 92 92 92 171 171 171 F F F 1 020-039 85 85 85 161 161 161 F F F 2 040-059 94 94 94 176 176 176 F F F 3 060-079 88 88 88 164 164 164 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 90 90 90 168 168 168 F F F 6 120-139 94 94 94 174 174 174 F F F 7 140-159 94 94 94 174 174 174 F F F 8 160-179 92 92 92 173 173 173 F F F 9 180-199 88 88 88 165 165 165 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 91 91 91 170 170 170 F F F 12 240-259 94 94 94 174 174 174 F F F 13 260-279 91 91 91 170 170 170 F F F 14 280-299 91 91 91 170 170 170 F F F 15 300-319 91 91 91 170 170 170 F F F 16 320-339 91 91 91 170 170 170 F F F 17 340-359 94 94 94 177 177 177 F F F 18 360-379 94 94 94 177 177 177 F F F 19 380-399 86 86 86 162 162 162 T T T 20 400-419 94 94 94 176 176 176 F F F 21 420-439 94 94 94 176 176 176 F F F 22 440-459 94 94 94 176 176 176 F F F 23 460-479 94 94 94 177 177 177 F F F 24 480-499 92 92 92 171 171 171 F F F 25 500-519 87 87 87 163 163 163 F F F 26 520-539 90 90 90 168 168 168 F F F 27 540-559 88 88 88 166 166 166 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 92 92 92 172 172 172 F F F Max gen: 94, Max flush: 177 A Difference in extent metadata is expected here On loop 47 Verifying data now No change after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 92 92 92 171 171 171 F F F 1 020-039 85 85 85 161 161 161 F F F 2 040-059 94 94 94 176 176 176 F F F 3 060-079 88 88 88 164 164 164 F F F 4 080-099 85 85 85 161 161 161 F F F 5 100-119 90 90 90 168 168 168 F F F 6 120-139 94 94 94 174 174 174 F F F 7 140-159 94 94 94 174 174 174 F F F 8 160-179 92 92 92 173 173 173 F F F 9 180-199 88 88 88 165 165 165 F F F 10 200-219 80 80 80 149 149 149 F F F 11 220-239 91 91 91 170 170 170 F F F 12 240-259 94 94 94 174 174 174 F F F 13 260-279 91 91 91 170 170 170 F F F 14 280-299 91 91 91 170 170 170 F F F 15 300-319 91 91 91 170 170 170 F F F 16 320-339 91 91 91 170 170 170 F F F 17 340-359 94 94 94 177 177 177 F F F 18 360-379 94 94 94 177 177 177 F F F 19 380-399 95 95 95 178 178 178 F F F 20 400-419 94 94 94 176 176 176 F F F 21 420-439 94 94 94 176 176 176 F F F 22 440-459 94 94 94 176 176 176 F F F 23 460-479 94 94 94 177 177 177 F F F 24 480-499 92 92 92 171 171 171 F F F 25 500-519 87 87 87 163 163 163 F F F 26 520-539 90 90 90 168 168 168 F F F 27 540-559 88 88 88 166 166 166 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 92 92 92 172 172 172 F F F Max gen: 95, Max flush: 178 Begin loop 48 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.922755468Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923300044Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923331071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569} {"msg":"Crucible a34b0b24-d141-4478-8294-90b71afadee4 has session id: f8168561-82ee-4ba5-8a3e-046d6fe76851","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923461418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Upstairs opts: Upstairs UUID: a34b0b24-d141-4478-8294-90b71afadee4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923503374Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Crucible stats registered with UUID: a34b0b24-d141-4478-8294-90b71afadee4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923527001Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923827682Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569} {"msg":"a34b0b24-d141-4478-8294-90b71afadee4 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923930511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923949949Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923965448Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923979416Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"1","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.923993075Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"1","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.924006674Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"2","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.924021142Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"2","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.924067848Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"io task","client":"2","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.924085736Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"io task","client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.924099784Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"io task","client":"1","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:06.924270298Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"io task","client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"ds_connection connected from Ok(127.0.0.1:46685)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.924307144Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"io task","client":"2","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:06.924339421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"io task","client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:06.924410334Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.924439241Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.924465308Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"ds_connection connected from Ok(127.0.0.1:43065)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.924492156Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"io task","client":"1","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.92454783Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"io task","client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.925138732Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"1","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.925164619Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"1","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.925179488Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"2","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.925193756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"2","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.925764059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"2","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.925900696Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"1","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.926255261Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"2","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.926283318Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"2","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.926302266Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.926385288Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"1","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.926404156Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"1","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:06.926418265Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.92571827Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"io task","client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"ds_connection connected from Ok(127.0.0.1:33446)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.925936138Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"io task","client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.927180385Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.927213112Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.927775696Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928390275Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928413673Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[0]R flush_numbers[0..12]: [171, 161, 176, 164, 161, 168, 174, 174, 173, 165, 149, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928431581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[0]R generation[0..12]: [92, 85, 94, 88, 85, 90, 94, 94, 92, 88, 80, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928449149Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928467118Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[1]R flush_numbers[0..12]: [171, 161, 176, 164, 161, 168, 174, 174, 173, 165, 149, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928485206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[1]R generation[0..12]: [92, 85, 94, 88, 85, 90, 94, 94, 92, 88, 80, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928507234Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928525382Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[2]R flush_numbers[0..12]: [171, 161, 176, 164, 161, 168, 174, 174, 173, 165, 149, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.92854298Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[2]R generation[0..12]: [92, 85, 94, 88, 85, 90, 94, 94, 92, 88, 80, 91]",Wait for a query_work_queue command to finish before sending IO "v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928564328Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928582536Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928599185Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928616053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Next flush: 179","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928632411Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.92864883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"":"downstairs","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928665368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928680486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"0","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928697255Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"1","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928714053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"client":"2","":"downstairs","session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928731172Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.92874801Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"a34b0b24-d141-4478-8294-90b71afadee4 is now active with session: f8168561-82ee-4ba5-8a3e-046d6fe76851","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928763208Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:16.928775667Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569,"session_id":"f8168561-82ee-4ba5-8a3e-046d6fe76851"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2024-07-09T21:42:16.928788286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 182 len 28672 data: 5 6 4 4 4 6 5 02/30 Read block 304 len 28672 03/30 Write block 506 len 40960 data: 8 8 9 7 9 9 10 8 8 7 04/30 Read block 520 len 20480 05/30 Read block 95 len 40960 06/30 Write block 505 len 4096 data: 7 07/30 Read block 529 len 16384 08/30 Read block 50 len 12288 09/30 Write block 256 len 12288 data: 6 5 5 10/30 Write block 72 len 36864 data: 8 7 7 5 6 6 7 6 6 11/30 Read block 38 len 4096 12/30 Write block 443 len 24576 data: 10 10 10 9 10 10 13/30 Read block 183 len 36864 14/30 Write block 506 len 4096 data: 9 15/30 Read block 144 len 12288 16/30 Read block 540 len 40960 17/30 Read block 489 len 40960 18/30 Read block 142 len 28672 19/30 Write block 551 len 28672 data: 11 11 9 10 9 11 11 20/30 Read block 212 len 20480 21/30 Write block 209 len 24576 data: 11 10 11 11 11 9 22/30 Read block 295 len 8192 23/30 Write block 82 len 4096 data: 7 24/30 Write block 23 len 16384 data: 5 5 6 7 25/30 Write block 66 len 36864 data: 7 6 6 6 8 10 9 8 8 26/30 Write block 62 len 32768 data: 10 8 6 6 8 7 7 7 27/30 Write block 587 len 28672 data: 10 10 11 11 11 8 9 28/30 Write block 468 len 36864 data: 9 9 8 6 7 7 6 4 4 29/30 Read block 239 len 28672 30/30 Write block 151 len 12288 data: 6 6 6 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 6 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Write 1 Done Done Done false 25 Acked 1024 Write 4 Done Done Done false 26 Acked 1025 Write 9 Done Done Done false 27 Acked 1026 Write 8 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Write 9 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 9 9 9 27 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:16.953270919Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1569} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 92 92 92 171 171 171 F F F 1 020-039 85 85 85 161 161 161 T T T 2 040-059 94 94 94 176 176 176 F F F 3 060-079 96 96 96 179 179 179 T T T 4 080-099 96 96 96 179 179 179 T T T 5 100-119 90 90 90 168 168 168 F F F 6 120-139 94 94 94 174 174 174 F F F 7 140-159 94 94 94 174 174 174 T T T 8 160-179 92 92 92 173 173 173 F F F 9 180-199 96 96 96 179 179 179 F F F 10 200-219 80 80 80 149 149 149 T T T 11 220-239 91 91 91 170 170 170 F F F 12 240-259 96 96 96 179 179 179 F F F 13 260-279 91 91 91 170 170 170 F F F 14 280-299 91 91 91 170 170 170 F F F 15 300-319 91 91 91 170 170 170 F F F 16 320-339 91 91 91 170 170 170 F F F 17 340-359 94 94 94 177 177 177 F F F 18 360-379 94 94 94 177 177 177 F F F 19 380-399 95 95 95 178 178 178 F F F 20 400-419 94 94 94 176 176 176 F F F 21 420-439 94 94 94 176 176 176 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 94 94 94 177 177 177 T T T 24 480-499 92 92 92 171 171 171 F F F 25 500-519 96 96 96 179 179 179 F F F 26 520-539 90 90 90 168 168 168 F F F 27 540-559 96 96 96 179 179 179 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 92 92 92 172 172 172 T T T Max gen: 96, Max flush: 179 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 92 92 92 171 171 171 F F F 1 020-039 97 97 97 180 180 180 F F F 2 040-059 94 94 94 176 176 176 F F F 3 060-079 97 97 97 180 180 180 F F F 4 080-099 97 97 97 180 180 180 F F F 5 100-119 90 90 90 168 168 168 F F F 6 120-139 94 94 94 174 174 174 F F F 7 140-159 97 97 97 180 180 180 F F F 8 160-179 92 92 92 173 173 173 F F F 9 180-199 96 96 96 179 179 179 F F F 10 200-219 97 97 97 180 180 180 F F F 11 220-239 91 91 91 170 170 170 F F F 12 240-259 96 96 96 179 179 179 F F F 13 260-279 91 91 91 170 170 170 F F F 14 280-299 91 91 91 170 170 170 F F F 15 300-319 91 91 91 170 170 170 F F F 16 320-339 91 91 91 170 170 170 F F F 17 340-359 94 94 94 177 177 177 F F F 18 360-379 94 94 94 177 177 177 F F F 19 380-399 95 95 95 178 178 178 F F F 20 400-419 94 94 94 176 176 176 F F F 21 420-439 94 94 94 176 176 176 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 97 97 97 180 180 180 F F F 24 480-499 92 92 92 171 171 171 F F F 25 500-519 96 96 96 179 179 179 F F F 26 520-539 90 90 90 168 168 168 F F F 27 540-559 96 96 96 179 179 179 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 97 97 97 180 180 180 F F F Max gen: 97, Max flush: 180 Begin loop 49 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.203523081Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.203810382Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.20383011Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577} {"msg":"Crucible f44de8e8-96b6-4728-a31c-dbca4627bf20 has session id: 0db3250c-9b6a-4932-9e4f-26fa61d955d8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204331471Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Upstairs opts: Upstairs UUID: f44de8e8-96b6-4728-a31c-dbca4627bf20, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204372017Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Crucible stats registered with UUID: f44de8e8-96b6-4728-a31c-dbca4627bf20","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204389545Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204749179Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577} {"msg":"f44de8e8-96b6-4728-a31c-dbca4627bf20 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204779896Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204794105Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204818112Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204832361Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"1","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.20484591Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"1","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204859638Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"2","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204873507Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"2","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204894265Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"io task","client":"2","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204908334Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"io task","client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.204921962Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"io task","client":"1","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:31.205099684Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"io task","client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:31.205126482Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"io task","client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:31.205154549Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.205176917Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.205199895Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"ds_connection connected from Ok(127.0.0.1:44986)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.205222192Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"io task","client":"2","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"ds_connection connected from Ok(127.0.0.1:61382)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.20524608Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"io task","client":"1","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.205430342Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"io task","client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.205963769Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"2","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.205991506Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"2","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.206007645Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"1","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.206021583Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"1","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.206630933Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"1","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.20665783Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"2","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.207174969Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"2","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.207203846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"2","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.207218825Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.207231393Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"1","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.207244852Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"1","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:31.207258431Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.206431638Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"io task","client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"ds_connection connected from Ok(127.0.0.1:56753)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.206641017Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"io task","client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.208474245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.208508992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209143139Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209683995Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209711742Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[0]R flush_numbers[0..12]: [171, 180, 176, 180, 180, 168, 174, 180, 173, 179, 180, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209728341Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[0]R generation[0..12]: [92, 97, 94, 97, 97, 90, 94, 97, 92, 96, 97, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209743309Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209757948Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[1]R flush_numbers[0..12]: [171, 180, 176, 180, 180, 168, 174, 180, 173, 179, 180, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209775566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[1]R generation[0..12]: [92, 97, 94, 97, 97, 90, 94, 97, 92, 96, 97, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209790065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209804443Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[2]R flush_numbers[0..12]: [171, 180, 176, 180, 180, 168, 174, 180, 173, 179, 180, 170]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209818632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[2]R generation[0..12]: [92, 97, 94, 97, 97, 90, 94, 97, 92, 96, 97, 91]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209832461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2024-07-09T21:42:41.209846709Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209864367Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209878006Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Next flush: 181","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209891315Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209904453Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"":"downstairs","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209917922Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209930131Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"0","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209943439Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"1","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209956658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"client":"2","":"downstairs","session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209970137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.209984195Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"f44de8e8-96b6-4728-a31c-dbca4627bf20 is now active with session: 0db3250c-9b6a-4932-9e4f-26fa61d955d8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.210011903Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.210025801Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577,"session_id":"0db3250c-9b6a-4932-9e4f-26fa61d955d8"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:41.21003838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 263 len 12288 data: 9 8 9 02/30 Write block 216 len 8192 data: 8 9 03/30 Write block 44 len 8192 data: 6 6 04/30 Read block 355 len 28672 05/30 Read block 297 len 40960 06/30 Read block 265 len 16384 07/30 Write block 162 len 8192 data: 6 7 08/30 Write block 289 len 28672 data: 11 8 8 7 6 7 8 09/30 Write block 137 len 24576 data: 7 7 9 8 6 7 10/30 Flush 11/30 Write block 384 len 40960 data: 6 9 10 10 12 14 13 15 14 13 12/30 Write block 75 len 12288 data: 6 7 7 13/30 Read block 437 len 36864 14/30 Write block 526 len 4096 data: 9 15/30 Write block 221 len 12288 data: 8 10 8 16/30 Write block 99 len 12288 data: 10 10 9 17/30 Read block 518 len 20480 18/30 Flush 19/30 Write block 241 len 20480 data: 5 5 5 5 4 20/30 Write block 339 len 8192 data: 6 6 21/30 Read block 5 len 24576 22/30 Write block 61 len 24576 data: 11 11 9 7 7 9 23/30 Write block 113 len 20480 data: 11 12 11 9 9 24/30 Write block 80 len 16384 data: 7 6 8 8 25/30 Read block 134 len 36864 26/30 Read block 78 len 36864 27/30 Write block 160 len 36864 data: 9 7 7 8 7 8 8 7 5 28/30 Write block 386 len 40960 data: 11 11 13 15 14 16 15 14 14 13 29/30 Read block 551 len 40960 30/30 Write block 141 len 20480 data: 7 8 8 7 6 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 21 Acked 1020 Write 2 Done Done Done false 22 Acked 1021 Read 6 Done Done Done false 23 Acked 1022 Write 6 Done Done Done false 24 Acked 1023 Write 5 Done Done Done false 25 Acked 1024 Write 4 Done Done Done false 26 Acked 1025 Read 9 Done Done Done false 27 Acked 1026 Read 9 Done Done Done false 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Write 10 Done Done Done false 30 Acked 1029 Read 10 Done Done Done false 31 Acked 1030 Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 10 10 10 30 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1019 1019 1019 Downstairs last five completed: 1019 1018 1017 1016 1015 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:41.237323286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1577} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 92 92 92 171 171 171 F F F 1 020-039 97 97 97 180 180 180 F F F 2 040-059 98 98 98 181 181 181 F F F 3 060-079 98 98 98 182 182 182 T T T 4 080-099 98 98 98 182 182 182 T T T 5 100-119 98 98 98 182 182 182 T T T 6 120-139 98 98 98 181 181 181 F F F 7 140-159 98 98 98 181 181 181 T T T 8 160-179 98 98 98 181 181 181 T T T 9 180-199 96 96 96 179 179 179 F F F 10 200-219 98 98 98 181 181 181 F F F 11 220-239 98 98 98 182 182 182 F F F 12 240-259 98 98 98 183 183 183 F F F 13 260-279 98 98 98 181 181 181 F F F 14 280-299 98 98 98 181 181 181 F F F 15 300-319 91 91 91 170 170 170 F F F 16 320-339 91 91 91 170 170 170 T T T 17 340-359 94 94 94 177 177 177 T T T 18 360-379 94 94 94 177 177 177 F F F 19 380-399 98 98 98 182 182 182 T T T 20 400-419 94 94 94 176 176 176 F F F 21 420-439 94 94 94 176 176 176 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 97 97 97 180 180 180 F F F 24 480-499 92 92 92 171 171 171 F F F 25 500-519 96 96 96 179 179 179 F F F 26 520-539 98 98 98 182 182 182 F F F 27 540-559 96 96 96 179 179 179 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 97 97 97 180 180 180 F F F Max gen: 98, Max flush: 183 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 92 92 92 171 171 171 F F F 1 020-039 97 97 97 180 180 180 F F F 2 040-059 98 98 98 181 181 181 F F F 3 060-079 99 99 99 184 184 184 F F F 4 080-099 99 99 99 184 184 184 F F F 5 100-119 99 99 99 184 184 184 F F F 6 120-139 98 98 98 181 181 181 F F F 7 140-159 99 99 99 184 184 184 F F F 8 160-179 99 99 99 184 184 184 F F F 9 180-199 96 96 96 179 179 179 F F F 10 200-219 98 98 98 181 181 181 F F F 11 220-239 98 98 98 182 182 182 F F F 12 240-259 98 98 98 183 183 183 F F F 13 260-279 98 98 98 181 181 181 F F F 14 280-299 98 98 98 181 181 181 F F F 15 300-319 91 91 91 170 170 170 F F F 16 320-339 99 99 99 184 184 184 F F F 17 340-359 99 99 99 184 184 184 F F F 18 360-379 94 94 94 177 177 177 F F F 19 380-399 99 99 99 184 184 184 F F F 20 400-419 94 94 94 176 176 176 F F F 21 420-439 94 94 94 176 176 176 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 97 97 97 180 180 180 F F F 24 480-499 92 92 92 171 171 171 F F F 25 500-519 96 96 96 179 179 179 F F F 26 520-539 98 98 98 182 182 182 F F F 27 540-559 96 96 96 179 179 179 F F F 28 560-579 84 84 84 158 158 158 F F F 29 580-599 97 97 97 180 180 180 F F F Max gen: 99, Max flush: 184 Begin loop 50 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.50198906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.502644315Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.502684211Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585} {"msg":"Crucible c89ef883-39f3-4ce2-9060-3b68a01d7979 has session id: f12d612d-970b-45b8-840a-f50cb15b08ff","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.502709978Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Upstairs opts: Upstairs UUID: c89ef883-39f3-4ce2-9060-3b68a01d7979, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.502748405Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Crucible stats registered with UUID: c89ef883-39f3-4ce2-9060-3b68a01d7979","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.502774732Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503122028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585} {"msg":"c89ef883-39f3-4ce2-9060-3b68a01d7979 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503185221Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503210239Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"0","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503226847Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"0","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503241136Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"1","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503254824Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"1","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503275052Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.50330066Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503315588Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"io task","client":"0","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503329747Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"io task","client":"1","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503349125Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"io task","client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:64070)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503486381Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"io task","client":"1","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:56675)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503520098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"io task","client":"0","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:55.503709379Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"io task","client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:55.503749445Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"io task","client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:42:55.503863864Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.503889321Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.50390559Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.504011109Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"io task","client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.504497461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"0","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.504551196Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"0","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.504582183Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"1","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.50460841Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"1","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.505139428Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"1","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.505174644Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"0","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.505681874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"1","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.505709091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"1","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.505724079Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.505736888Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"0","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.505750917Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"0","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:42:55.505764246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.504597446Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"io task","client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:47094)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.504816395Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"io task","client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.506169141Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.506205247Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.506700638Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507208158Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507231805Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[0]R flush_numbers[0..12]: [171, 180, 181, 184, 184, 184, 181, 184, 184, 179, 181, 182]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507250114Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[0]R generation[0..12]: [92, 97, 98, 99, 99, 99, 98, 99, 99, 96, 98, 98]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507273031Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507291459Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[1]R flush_numbers[0..12]: [171, 180, 181, 184, 184, 184, 181, 184, 184, 179, 181, 182]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507308658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[1]R generation[0..12]: [92, 97, 98, 99, 99, 99, 98, 99, 99, 96, 98, 98]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507325846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507343744Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[2]R flush_numbers[0..12]: [171, 180, 181, 184, 184, 184, 181, 184, 184, 179, 181, 182]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507361223Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[2]R generation[0..12]: [92, 97, 98, 99, 99, 99, 98, 99, 99, 96, 98, 98]","v"Wait for a query_work_queue command to finish before sending IO :0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507381421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507399539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507415907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507432576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Next flush: 185","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507448674Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507464912Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"":"downstairs","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507481591Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507496779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"0","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507513308Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"1","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507532066Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"client":"2","":"downstairs","session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507548864Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507564182Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"c89ef883-39f3-4ce2-9060-3b68a01d7979 is now active with session: f12d612d-970b-45b8-840a-f50cb15b08ff","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507580491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507595669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585,"session_id":"f12d612d-970b-45b8-840a-f50cb15b08ff"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:05.507611118Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 39 len 36864 data: 9 6 4 5 7 7 7 6 6 02/30 Write block 553 len 32768 data: 10 11 10 12 12 12 10 10 03/30 Write block 317 len 8192 data: 8 9 04/30 Read block 327 len 16384 05/30 Read block 307 len 12288 06/30 Read block 317 len 8192 07/30 Read block 465 len 28672 08/30 Read block 186 len 8192 09/30 Write block 569 len 24576 data: 9 10 12 11 11 11 10/30 Read block 270 len 20480 11/30 Write block 385 len 40960 data: 10 12 12 14 16 15 17 16 15 15 12/30 Read block 565 len 20480 13/30 Read block 125 len 40960 14/30 Read block 470 len 36864 15/30 Read block 394 len 32768 16/30 Write block 407 len 36864 data: 9 11 11 10 8 9 7 7 8 17/30 Write block 311 len 36864 data: 8 9 9 10 7 7 9 10 10 18/30 Write block 462 len 12288 data: 11 10 8 19/30 Flush 20/30 Write block 211 len 28672 data: 12 12 12 10 10 9 10 21/30 Write block 137 len 12288 data: 8 8 10 22/30 Write block 124 len 12288 data: 11 11 11 23/30 Write block 487 len 12288 data: 10 10 10 24/30 Read block 233 len 40960 25/30 Write block 56 len 12288 data: 13 13 13 26/30 Write block 182 len 32768 data: 6 7 5 5 5 7 6 7 27/30 Write block 233 len 24576 data: 8 9 8 8 6 7 28/30 Write block 193 len 8192 data: 11 11 29/30 Write block 120 len 8192 data: 4 5 30/30 Read block 336 len 20480 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 7 Done Done Done false 21 Acked 1020 Write 3 Done Done Done false 22 Acked 1021 Write 3 Done Done Done false 23 Acked 1022 Write 3 Done Done Done false 24 Acked 1023 Read 10 Done Done Done false 25 Acked 1024 Write 3 Done Done Done false 26 Acked 1025 Write 8 Done Done Done false 27 Acked 1026 Write 6 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Write 2 Done Done Sent false 30 Acked 1029 Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 2 3 Done 11 10 9 30 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:43:05.532923609Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1585} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 92 92 92 171 171 171 F F F 1 020-039 100 100 100 185 185 185 F F F 2 040-059 100 100 100 185 185 185 T T T 3 060-079 99 99 99 184 184 184 F F F 4 080-099 99 99 99 184 184 184 F F F 5 100-119 99 99 99 184 184 184 F F F 6 120-139 98 98 98 181 181 181 T T T 7 140-159 99 99 99 184 184 184 F F F 8 160-179 99 99 99 184 184 184 F F F 9 180-199 96 96 96 179 179 179 T T T 10 200-219 98 98 98 181 181 181 T T T 11 220-239 98 98 98 182 182 182 T T T 12 240-259 98 98 98 183 183 183 F F F 13 260-279 98 98 98 181 181 181 F F F 14 280-299 98 98 98 181 181 181 F F F 15 300-319 100 100 100 185 185 185 F F F 16 320-339 99 99 99 184 184 184 F F F 17 340-359 99 99 99 184 184 184 F F F 18 360-379 94 94 94 177 177 177 F F F 19 380-399 100 100 100 185 185 185 F F F 20 400-419 100 100 100 185 185 185 F F F 21 420-439 94 94 94 176 176 176 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 100 100 100 185 185 185 F F F 24 480-499 92 92 92 171 171 171 T T T 25 500-519 96 96 96 179 179 179 F F F 26 520-539 98 98 98 182 182 182 F F F 27 540-559 100 100 100 185 185 185 F F F 28 560-579 100 100 100 185 185 185 F F F 29 580-599 97 97 97 180 180 180 F F F Max gen: 100, Max flush: 185 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 92 92 92 171 171 171 F F F 1 020-039 100 100 100 185 185 185 F F F 2 040-059 101 101 101 186 186 186 F F F 3 060-079 99 99 99 184 184 184 F F F 4 080-099 99 99 99 184 184 184 F F F 5 100-119 99 99 99 184 184 184 F F F 6 120-139 101 101 101 186 186 186 F F F 7 140-159 99 99 99 184 184 184 F F F 8 160-179 99 99 99 184 184 184 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 101 101 101 186 186 186 F F F 11 220-239 101 101 101 186 186 186 F F F 12 240-259 98 98 98 183 183 183 F F F 13 260-279 98 98 98 181 181 181 F F F 14 280-299 98 98 98 181 181 181 F F F 15 300-319 100 100 100 185 185 185 F F F 16 320-339 99 99 99 184 184 184 F F F 17 340-359 99 99 99 184 184 184 F F F 18 360-379 94 94 94 177 177 177 F F F 19 380-399 100 100 100 185 185 185 F F F 20 400-419 100 100 100 185 185 185 F F F 21 420-439 94 94 94 176 176 176 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 100 100 100 185 185 185 F F F 24 480-499 101 101 101 186 186 186 F F F 25 500-519 96 96 96 179 179 179 F F F 26 520-539 98 98 98 182 182 182 F F F 27 540-559 100 100 100 185 185 185 F F F 28 560-579 100 100 100 185 185 185 F F F 29 580-599 97 97 97 180 180 180 F F F Max gen: 101, Max flush: 186 Begin loop 51 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.778970647Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.779277627Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.779306914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593} {"msg":"Crucible 61edcaaa-556e-4c99-8c26-36fc8c87e30b has session id: 7d5f6cd1-afa9-4c94-bd9e-87b25a80962c","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.779679887Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Upstairs opts: Upstairs UUID: 61edcaaa-556e-4c99-8c26-36fc8c87e30b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.779726372Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Crucible stats registered with UUID: 61edcaaa-556e-4c99-8c26-36fc8c87e30b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.77975304Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780076128Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593} {"msg":"61edcaaa-556e-4c99-8c26-36fc8c87e30b active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780105075Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780120133Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"0","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780135022Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"0","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780149011Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"1","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780162809Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"1","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780175988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780195136Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780235162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"io task","client":"0","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.78025653Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"io task","client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780271038Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"io task","client":"1","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:43:19.780440852Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"io task","client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"ds_connection connected from Ok(127.0.0.1:33264)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780477618Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"io task","client":"1","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:43:19.780505565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"io task","client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:43:19.780533023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.78055582Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780579618Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"ds_connection connected from Ok(127.0.0.1:51556)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780601926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"io task","client":"0","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.780700496Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"io task","client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.781231273Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"0","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.78126875Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"0","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.781298407Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"1","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.781329504Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"1","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.781803087Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"0","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.781930044Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"1","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.782241503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"0","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.782311296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"0","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.782343713Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.782440314Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"1","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.782463991Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"1","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:19.78247889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.781376404Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"io task","client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"ds_connection connected from Ok(127.0.0.1:49987)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.781560895Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"io task","client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.78262973Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.782656707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783178245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783744099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783767037Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[0]R flush_numbers[0..12]: [171, 185, 186, 184, 184, 184, 186, 184, 184, 186, 186, 186]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783787285Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[0]R generation[0..12]: [92, 100, 101, 99, 99, 99, 101, 99, 99, 101, 101, 101]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783803333Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783818062Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[1]R flush_numbers[0..12]: [171, 185, 186, 184, 184, 184, 186, 184, 184, 186, 186, 186]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.78383237Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[1]R generation[0..12]: [92, 100, 101, 99, 99, 99, 101, 99, 99, 101, 101, 101]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783846499Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783860648Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[2]R flush_numbers[0..12]: [171, 185, 186, 184, 184, 184, 186, 184, 184, 186, 186, 186]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783874566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[2]R generation[0..12]: [92, 100, 101, 99, 99, 99, 101, 99, 99, 101, 101, 101]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783888255Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783902353Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783915252Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6GWait for a query_work_queue command to finish before sending IO ","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.78393283Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Next flush: 187","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783946049Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783959138Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"":"downstairs","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783972416Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783986355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"0","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.783999914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"1","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.784012992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"client":"2","":"downstairs","session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.784026311Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.78403873Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"61edcaaa-556e-4c99-8c26-36fc8c87e30b is now active with session: 7d5f6cd1-afa9-4c94-bd9e-87b25a80962c","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.784051669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.784063537Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593,"session_id":"7d5f6cd1-afa9-4c94-bd9e-87b25a80962c"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:29.784075876Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 517 len 16384 data: 7 6 5 5 02/30 Read block 266 len 40960 03/30 Read block 102 len 32768 04/30 Write block 200 len 36864 data: 9 8 8 8 9 11 11 13 13 05/30 Read block 455 len 4096 06/30 Write block 147 len 16384 data: 6 5 5 5 07/30 Write block 334 len 20480 data: 7 7 5 6 5 08/30 Write block 330 len 36864 data: 8 10 9 8 8 8 6 7 6 09/30 Write block 32 len 24576 data: 11 10 9 11 11 14 10/30 Read block 460 len 12288 11/30 Write block 265 len 12288 data: 10 9 8 12/30 Read block 272 len 4096 13/30 Flush 14/30 Write block 10 len 36864 data: 7 7 9 11 12 12 13 12 9 15/30 Flush 16/30 Write block 6 len 40960 data: 3 5 6 7 8 8 10 12 13 13 17/30 Write block 62 len 8192 data: 12 10 18/30 Read block 510 len 24576 19/30 Write block 107 len 12288 data: 8 10 10 20/30 Write block 517 len 16384 data: 8 7 6 6 21/30 Write block 586 len 36864 data: 9 11 11 12 12 12 9 10 12 22/30 Read block 216 len 8192 23/30 Read block 136 len 40960 24/30 Read block 504 len 8192 25/30 Read block 268 len 40960 26/30 Read block 443 len 8192 27/30 Write block 312 len 32768 data: 10 10 11 8 8 10 11 11 28/30 Read block 158 len 24576 29/30 Read block 313 len 12288 30/30 Read block 30 len 12288 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 3 Done Done Done false 21 Acked 1020 Write 4 Done Done Done false 22 Acked 1021 Write 9 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Read 10 Done Done Done false 25 Acked 1024 Read 2 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Read 2 Done Done Done false 28 Acked 1027 Write 8 Done Done Done false 29 Acked 1028 Read 6 Done Done Done false 30 Acked 1029 Read 3 Done Done Done false 31 Acked 1030 Read 3 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 12 12 12 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:43:29.810947963Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1593} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 189 189 189 F F F 1 020-039 102 102 102 187 187 187 F F F 2 040-059 101 101 101 186 186 186 F F F 3 060-079 102 102 102 189 189 189 F F F 4 080-099 99 99 99 184 184 184 F F F 5 100-119 99 99 99 184 184 184 T T T 6 120-139 101 101 101 186 186 186 F F F 7 140-159 102 102 102 187 187 187 F F F 8 160-179 99 99 99 184 184 184 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 102 102 102 187 187 187 F F F 11 220-239 101 101 101 186 186 186 F F F 12 240-259 98 98 98 183 183 183 F F F 13 260-279 102 102 102 187 187 187 F F F 14 280-299 98 98 98 181 181 181 F F F 15 300-319 100 100 100 185 185 185 T T T 16 320-339 102 102 102 187 187 187 F F F 17 340-359 99 99 99 184 184 184 F F F 18 360-379 94 94 94 177 177 177 F F F 19 380-399 100 100 100 185 185 185 F F F 20 400-419 100 100 100 185 185 185 F F F 21 420-439 94 94 94 176 176 176 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 100 100 100 185 185 185 F F F 24 480-499 101 101 101 186 186 186 F F F 25 500-519 102 102 102 187 187 187 T T T 26 520-539 102 102 102 187 187 187 T T T 27 540-559 100 100 100 185 185 185 F F F 28 560-579 100 100 100 185 185 185 F F F 29 580-599 97 97 97 180 180 180 T T T Max gen: 102, Max flush: 189 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 189 189 189 F F F 1 020-039 102 102 102 187 187 187 F F F 2 040-059 101 101 101 186 186 186 F F F 3 060-079 102 102 102 189 189 189 F F F 4 080-099 99 99 99 184 184 184 F F F 5 100-119 103 103 103 190 190 190 F F F 6 120-139 101 101 101 186 186 186 F F F 7 140-159 102 102 102 187 187 187 F F F 8 160-179 99 99 99 184 184 184 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 102 102 102 187 187 187 F F F 11 220-239 101 101 101 186 186 186 F F F 12 240-259 98 98 98 183 183 183 F F F 13 260-279 102 102 102 187 187 187 F F F 14 280-299 98 98 98 181 181 181 F F F 15 300-319 103 103 103 190 190 190 F F F 16 320-339 102 102 102 187 187 187 F F F 17 340-359 99 99 99 184 184 184 F F F 18 360-379 94 94 94 177 177 177 F F F 19 380-399 100 100 100 185 185 185 F F F 20 400-419 100 100 100 185 185 185 F F F 21 420-439 94 94 94 176 176 176 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 100 100 100 185 185 185 F F F 24 480-499 101 101 101 186 186 186 F F F 25 500-519 103 103 103 190 190 190 F F F 26 520-539 103 103 103 190 190 190 F F F 27 540-559 100 100 100 185 185 185 F F F 28 560-579 100 100 100 185 185 185 F F F 29 580-599 103 103 103 190 190 190 F F F Max gen: 103, Max flush: 190 Begin loop 52 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.039797146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.040405736Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.040435433Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601} {"msg":"Crucible ef6892c6-1098-475d-84e3-5763a4bfa858 has session id: 0ba57f77-3b80-494c-9eef-37965fcdac85","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.040457011Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Upstairs opts: Upstairs UUID: ef6892c6-1098-475d-84e3-5763a4bfa858, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.040491938Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Crucible stats registered with UUID: ef6892c6-1098-475d-84e3-5763a4bfa858","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.040514415Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.040946643Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601} {"msg":"ef6892c6-1098-475d-84e3-5763a4bfa858 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.040981049Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041011416Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"0","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041028675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"0","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041042823Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"1","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041056542Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"1","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.04107327Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041087329Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041129505Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"io task","client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041146543Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"io task","client":"1","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041160481Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"io task","client":"0","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:43:44.041340384Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"io task","client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:43:44.04147174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"io task","client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ds_connection connected from Ok(127.0.0.1:46041)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041509257Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"io task","client":"1","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:43:44.041533374Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041547053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ds_connection connected from Ok(127.0.0.1:37045)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041568011Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"io task","client":"0","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041592099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.041650963Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"io task","client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.042242594Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"1","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.042271331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"1","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.04228693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"0","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.042301258Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"0","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.042897499Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"1","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.042924296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"0","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.043418198Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"1","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.043444205Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"1","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.043458914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.043471532Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"0","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.043485401Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"0","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:44.04349872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.042693315Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"io task","client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ds_connection connected from Ok(127.0.0.1:53822)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.042901054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"io task","client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.0439497Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.043984917Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.044506475Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045022494Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045041892Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[0]R flush_numbers[0..12]: [189, 187, 186, 189, 184, 190, 186, 187, 184, 186, 187, 186]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.04505764Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[0]R generation[0..12]: [102, 102, 101, 102, 99, 103, 101, 102, 99, 101, 102, 101]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045075249Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045093337Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[1]R flush_numbers[0..12]: [189, 187, 186, 189, 184, 190, 186, 187, 184, 186, 187, 186]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045111075Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[1]R generation[0..12]: [102, 102, 101, 102, 99, 103, 101, 102, 99, 101, 102, 101]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045128803Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045147042Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[2]R flush_numbers[0..12]: [189, 187, 186, 189, 184, 190, 186, 187, 184, 186, 187, 186]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.04516491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"[2]R generation[0..12]: [102, 102, 101, 102, 99, 103, 101, 102, 99, 101, 102, 101]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045182418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} Wait for a query_work_queue command to finish before sending IO {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045202976Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Max found gen is 104","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045220234Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Generation requested: 104 >= found:104","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045236963Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Next flush: 191","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045256601Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045273239Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"":"downstairs","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045290217Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045305496Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"0","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045322384Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"1","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045338992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"client":"2","":"downstairs","session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045355811Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045371459Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"ef6892c6-1098-475d-84e3-5763a4bfa858 is now active with session: 0ba57f77-3b80-494c-9eef-37965fcdac85","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045387958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045403276Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601,"session_id":"0ba57f77-3b80-494c-9eef-37965fcdac85"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:43:54.045418745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 255 len 4096 data: 6 02/30 Flush 03/30 Write block 144 len 16384 data: 8 7 8 7 04/30 Write block 300 len 32768 data: 11 10 8 7 8 10 10 9 05/30 Read block 252 len 16384 06/30 Write block 374 len 12288 data: 9 10 9 07/30 Write block 202 len 40960 data: 9 9 10 12 12 14 14 12 11 13 08/30 Read block 97 len 36864 09/30 Write block 69 len 8192 data: 8 9 10/30 Read block 245 len 16384 11/30 Write block 555 len 12288 data: 11 13 13 12/30 Read block 550 len 16384 13/30 Write block 165 len 16384 data: 9 9 8 6 14/30 Read block 466 len 12288 15/30 Write block 406 len 24576 data: 8 10 12 12 11 9 16/30 Flush 17/30 Write block 476 len 20480 data: 5 5 6 5 5 18/30 Read block 229 len 24576 19/30 Flush 20/30 Write block 545 len 24576 data: 6 8 8 9 9 11 21/30 Read block 466 len 20480 22/30 Read block 580 len 12288 23/30 Write block 298 len 36864 data: 9 9 12 11 9 8 9 11 11 24/30 Read block 519 len 32768 25/30 Write block 428 len 32768 data: 11 11 11 9 10 11 12 11 26/30 Read block 75 len 12288 27/30 Write block 237 len 8192 data: 7 8 28/30 Write block 571 len 4096 data: 13 29/30 Read block 389 len 20480 30/30 Read block 362 len 16384 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 6 Done Done Done false 21 Acked 1020 Read 5 Done Done Done false 22 Acked 1021 Read 3 Done Done Done false 23 Acked 1022 Write 9 Done Done Done false 24 Acked 1023 Read 8 Done Done Done false 25 Acked 1024 Write 8 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Write 1 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Read 4 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 11 11 11 33 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:43:54.07129163Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1601} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 189 189 189 F F F 1 020-039 102 102 102 187 187 187 F F F 2 040-059 101 101 101 186 186 186 F F F 3 060-079 104 104 104 192 192 192 F F F 4 080-099 99 99 99 184 184 184 F F F 5 100-119 103 103 103 190 190 190 F F F 6 120-139 101 101 101 186 186 186 F F F 7 140-159 104 104 104 192 192 192 F F F 8 160-179 104 104 104 192 192 192 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 104 104 104 192 192 192 F F F 11 220-239 101 101 101 186 186 186 T T T 12 240-259 104 104 104 191 191 191 F F F 13 260-279 102 102 102 187 187 187 F F F 14 280-299 98 98 98 181 181 181 T T T 15 300-319 104 104 104 192 192 192 T T T 16 320-339 102 102 102 187 187 187 F F F 17 340-359 99 99 99 184 184 184 F F F 18 360-379 104 104 104 192 192 192 F F F 19 380-399 100 100 100 185 185 185 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 94 94 94 176 176 176 T T T 22 440-459 96 96 96 179 179 179 F F F 23 460-479 104 104 104 193 193 193 F F F 24 480-499 104 104 104 193 193 193 F F F 25 500-519 103 103 103 190 190 190 F F F 26 520-539 103 103 103 190 190 190 F F F 27 540-559 104 104 104 192 192 192 T T T 28 560-579 100 100 100 185 185 185 T T T 29 580-599 103 103 103 190 190 190 F F F Max gen: 104, Max flush: 193 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 189 189 189 F F F 1 020-039 102 102 102 187 187 187 F F F 2 040-059 101 101 101 186 186 186 F F F 3 060-079 104 104 104 192 192 192 F F F 4 080-099 99 99 99 184 184 184 F F F 5 100-119 103 103 103 190 190 190 F F F 6 120-139 101 101 101 186 186 186 F F F 7 140-159 104 104 104 192 192 192 F F F 8 160-179 104 104 104 192 192 192 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 104 104 104 192 192 192 F F F 11 220-239 105 105 105 194 194 194 F F F 12 240-259 104 104 104 191 191 191 F F F 13 260-279 102 102 102 187 187 187 F F F 14 280-299 105 105 105 194 194 194 F F F 15 300-319 105 105 105 194 194 194 F F F 16 320-339 102 102 102 187 187 187 F F F 17 340-359 99 99 99 184 184 184 F F F 18 360-379 104 104 104 192 192 192 F F F 19 380-399 100 100 100 185 185 185 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 96 96 96 179 179 179 F F F 23 460-479 104 104 104 193 193 193 F F F 24 480-499 104 104 104 193 193 193 F F F 25 500-519 103 103 103 190 190 190 F F F 26 520-539 103 103 103 190 190 190 F F F 27 540-559 105 105 105 194 194 194 F F F 28 560-579 105 105 105 194 194 194 F F F 29 580-599 103 103 103 190 190 190 F F F Max gen: 105, Max flush: 194 Begin loop 53 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.319278305Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.319607993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.319628041Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609} {"msg":"Crucible dd27d261-4215-4b24-a54e-3af411aa2ec0 has session id: 6264ee02-ab82-472f-928e-102a1230824d","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320008143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Upstairs opts: Upstairs UUID: dd27d261-4215-4b24-a54e-3af411aa2ec0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320051989Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Crucible stats registered with UUID: dd27d261-4215-4b24-a54e-3af411aa2ec0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320076686Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320383446Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609} {"msg":"dd27d261-4215-4b24-a54e-3af411aa2ec0 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320535431Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320555059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320570057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320587476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"1","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320601524Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"1","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320623282Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"2","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.32064613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"2","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320660658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"io task","client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320674647Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"io task","client":"1","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320688615Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"io task","client":"2","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"ds_connection connected from Ok(127.0.0.1:35392)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.320939761Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"io task","client":"2","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:08.321059549Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"io task","client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"ds_connection connected from Ok(127.0.0.1:43316)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.321094845Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"io task","client":"1","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:08.321113993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"io task","client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:08.321131192Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.32115131Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.321166128Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.321271288Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"io task","client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.321814614Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"2","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.321840401Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"2","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.32185572Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"1","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.321870178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"1","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.322436192Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"2","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.32246289Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"1","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.322985318Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"2","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.323021384Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"2","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.323045672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.32306581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"1","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.323087018Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"1","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:08.323108076Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.322244628Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"io task","client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"ds_connection connected from Ok(127.0.0.1:43923)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.322376215Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"io task","client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.323754659Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.323790425Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.324366248Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.324926572Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.32494621Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[0]R flush_numbers[0..12]: [189, 187, 186, 192, 184, 190, 186, 192, 192, 186, 192, 194]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.324966188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[0]R generation[0..12]: [102, 102, 101, 104, 99, 103, 101, 104, 104, 101, 104, 105]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.324984837Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325003035Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[1]R flush_numbers[0..12]: [189, 187, 186, 192, 184, 190, 186, 192, 192, 186, 192, 194]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325020663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[1]R generation[0..12]: [102, 102, 101, 104, 99, 103, 101, 104, 104, 101, 104, 105]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325038201Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.32505579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[2]R flush_numbers[0..12]: [189, 187, 186, 192, 184, 190, 186, 192, 192, 186, 192, 194]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325073258Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[2]R generation[0..12]: [102, 102, 101, 104, 99, 103, 101, 104, 104, 101, 104, 105]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325090516Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:44:18.325108324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Max found gen is 106","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325126463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.32514737Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Next flush: 195","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325164109Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325180307Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"":"downstairs","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325196785Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325211654Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"0","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325228222Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"1","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325244761Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"client":"2","":"downstairs","session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325261469Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325276908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"dd27d261-4215-4b24-a54e-3af411aa2ec0 is now active with session: 6264ee02-ab82-472f-928e-102a1230824d","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325293106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325308145Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609,"session_id":"6264ee02-ab82-472f-928e-102a1230824d"} {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:18.325323373Z","hostname":"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 308 len 12288 data: 9 10 9 02/30 Read block 101 len 20480 03/30 Write block 40 len 40960 data: 7 5 6 8 8 8 7 7 6 6 04/30 Read block 534 len 16384 05/30 Write block 508 len 20480 data: 10 8 10 10 11 06/30 Flush 07/30 Write block 32 len 40960 data: 12 11 10 12 12 15 9 10 8 6 08/30 Read block 19 len 28672 09/30 Read block 431 len 12288 10/30 Read block 453 len 16384 11/30 Write block 441 len 16384 data: 11 11 11 11 12/30 Read block 508 len 20480 13/30 Read block 357 len 32768 14/30 Write block 59 len 28672 data: 11 11 12 13 11 8 8 15/30 Read block 322 len 40960 16/30 Write block 80 len 36864 data: 8 7 9 9 9 7 10 10 10 17/30 Read block 110 len 12288 18/30 Flush 19/30 Write block 219 len 36864 data: 7 6 9 11 9 8 7 7 5 20/30 Read block 205 len 40960 21/30 Read block 218 len 12288 22/30 Write block 68 len 32768 data: 8 9 10 11 10 9 9 7 23/30 Read block 206 len 20480 24/30 Flush 25/30 Write block 122 len 32768 data: 6 10 12 12 12 8 5 5 26/30 Read block 187 len 24576 27/30 Read block 144 len 12288 28/30 Write block 142 len 20480 data: 9 9 9 8 9 29/30 Read block 512 len 24576 30/30 Write block 369 len 8192 data: 11 9 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Write 8 Done Done Done false 27 Acked 1026 Read 6 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Write 5 Done Done Done false 30 Acked 1029 Read 6 Done Done Done false 31 Acked 1030 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:18.354189252Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1609} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 189 189 189 F F F 1 020-039 106 106 106 196 196 196 F F F 2 040-059 106 106 106 196 196 196 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 106 106 106 196 196 196 F F F 5 100-119 103 103 103 190 190 190 F F F 6 120-139 101 101 101 186 186 186 T T T 7 140-159 104 104 104 192 192 192 T T T 8 160-179 104 104 104 192 192 192 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 106 106 106 198 198 198 F F F 11 220-239 106 106 106 198 198 198 F F F 12 240-259 104 104 104 191 191 191 F F F 13 260-279 102 102 102 187 187 187 F F F 14 280-299 105 105 105 194 194 194 F F F 15 300-319 106 106 106 195 195 195 F F F 16 320-339 102 102 102 187 187 187 F F F 17 340-359 99 99 99 184 184 184 F F F 18 360-379 104 104 104 192 192 192 T T T 19 380-399 100 100 100 185 185 185 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 106 106 106 196 196 196 F F F 23 460-479 104 104 104 193 193 193 F F F 24 480-499 104 104 104 193 193 193 F F F 25 500-519 106 106 106 195 195 195 F F F 26 520-539 103 103 103 190 190 190 F F F 27 540-559 105 105 105 194 194 194 F F F 28 560-579 105 105 105 194 194 194 F F F 29 580-599 103 103 103 190 190 190 F F F Max gen: 106, Max flush: 198 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 189 189 189 F F F 1 020-039 106 106 106 196 196 196 F F F 2 040-059 106 106 106 196 196 196 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 106 106 106 196 196 196 F F F 5 100-119 103 103 103 190 190 190 F F F 6 120-139 107 107 107 199 199 199 F F F 7 140-159 107 107 107 199 199 199 F F F 8 160-179 104 104 104 192 192 192 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 106 106 106 198 198 198 F F F 11 220-239 106 106 106 198 198 198 F F F 12 240-259 104 104 104 191 191 191 F F F 13 260-279 102 102 102 187 187 187 F F F 14 280-299 105 105 105 194 194 194 F F F 15 300-319 106 106 106 195 195 195 F F F 16 320-339 102 102 102 187 187 187 F F F 17 340-359 99 99 99 184 184 184 F F F 18 360-379 107 107 107 199 199 199 F F F 19 380-399 100 100 100 185 185 185 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 106 106 106 196 196 196 F F F 23 460-479 104 104 104 193 193 193 F F F 24 480-499 104 104 104 193 193 193 F F F 25 500-519 106 106 106 195 195 195 F F F 26 520-539 103 103 103 190 190 190 F F F 27 540-559 105 105 105 194 194 194 F F F 28 560-579 105 105 105 194 194 194 F F F 29 580-599 103 103 103 190 190 190 F F F Max gen: 107, Max flush: 199 Begin loop 54 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.561367821Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.56178427Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.561809947Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617} {"msg":"Crucible 53526a78-50bc-479f-a9bc-caf6f0d57280 has session id: ee02dd34-424b-47a5-a2df-5e024d0f4a82","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.562243404Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Upstairs opts: Upstairs UUID: 53526a78-50bc-479f-a9bc-caf6f0d57280, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.56228874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Crucible stats registered with UUID: 53526a78-50bc-479f-a9bc-caf6f0d57280","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.562310088Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.562728816Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617} {"msg":"53526a78-50bc-479f-a9bc-caf6f0d57280 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.562797809Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.562823377Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"0","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.562839295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"0","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.562866892Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.56289353Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.562920547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"2","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.562946385Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"2","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.562971162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"io task","client":"0","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.56299456Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"io task","client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.563050314Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"io task","client":"2","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"ds_connection connected from Ok(127.0.0.1:33576)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.563076332Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"io task","client":"2","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"ds_connection connected from Ok(127.0.0.1:55727)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.563102039Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"io task","client":"0","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:32.563465133Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"io task","client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:32.56349397Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"io task","client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:32.563510929Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.563570733Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.56359678Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.563624467Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"io task","client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.563881742Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"2","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.563915649Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"2","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.563991331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"0","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.564017558Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"0","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.564454515Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"2","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.564578143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"0","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.564966004Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"2","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.564992802Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"2","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.565017309Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.565033718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"0","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.565047776Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"0","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:32.565060925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.5649646Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"io task","client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"ds_connection connected from Ok(127.0.0.1:54810)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.565153522Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"io task","client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.56618106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.566227705Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.566776891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567307728Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567336525Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[0]R flush_numbers[0..12]: [189, 196, 196, 198, 196, 190, 199, 199, 192, 186, 198, 198]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567355424Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[0]R generation[0..12]: [102, 106, 106, 106, 106, 103, 107, 107, 104, 101, 106, 106]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567373472Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.56739158Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[1]R flush_numbers[0..12]: [189, 196, 196, 198, 196, 190, 199, 199, 192, 186, 198, 198]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567409478Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[1]R generation[0..12]: [102, 106, 106, 106, 106, 103, 107, 107, 104, 101, 106, 106]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567426806Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567444584Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[2]R flush_numbers[0..12]: [189, 196, 196, 198, 196, 190, 199, 199, 192, 186, 198, 198]","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2024-07-09T21:44:42.567461893Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[2]R generation[0..12]: [102, 106, 106, 106, 106, 103, 107, 107, 104, 101, 106, 106]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567482501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567500139Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567516388Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567537415Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Next flush: 200","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567554254Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567570562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"":"downstairs","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567587401Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567602359Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"0","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567619147Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"1","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567635866Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"client":"2","":"downstairs","session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567652854Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567668153Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"53526a78-50bc-479f-a9bc-caf6f0d57280 is now active with session: ee02dd34-424b-47a5-a2df-5e024d0f4a82","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567684181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567698929Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617,"session_id":"ee02dd34-424b-47a5-a2df-5e024d0f4a82"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:42.567714178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 133 len 8192 data: 8 8 02/30 Read block 514 len 12288 03/30 Read block 348 len 16384 04/30 Read block 23 len 32768 05/30 Read block 50 len 16384 06/30 Write block 21 len 8192 data: 7 4 07/30 Read block 123 len 28672 08/30 Read block 166 len 40960 09/30 Read block 115 len 24576 10/30 Read block 389 len 40960 11/30 Read block 276 len 16384 12/30 Write block 164 len 8192 data: 8 10 13/30 Read block 355 len 16384 14/30 Read block 151 len 32768 15/30 Read block 534 len 24576 16/30 Write block 443 len 32768 data: 12 12 11 10 11 11 10 9 17/30 Read block 223 len 20480 18/30 Flush 19/30 Write block 246 len 40960 data: 4 3 5 3 3 4 4 4 4 7 20/30 Read block 550 len 32768 21/30 Write block 85 len 36864 data: 8 11 11 11 9 8 8 8 8 22/30 Read block 296 len 40960 23/30 Write block 92 len 12288 data: 9 9 9 24/30 Write block 329 len 24576 data: 8 9 11 10 9 9 25/30 Read block 336 len 40960 26/30 Read block 545 len 40960 27/30 Read block 498 len 4096 28/30 Read block 457 len 12288 29/30 Read block 595 len 20480 30/30 Write block 473 len 4096 data: 8 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 10 Done Done Done false 20 Acked 1019 Read 8 Done Done Done false 21 Acked 1020 Write 9 Done Done Done false 22 Acked 1021 Read 10 Done Done Done false 23 Acked 1022 Write 3 Done Done Done false 24 Acked 1023 Write 6 Done Done Done false 25 Acked 1024 Read 10 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Read 1 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 11 11 11 33 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:42.592075454Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1617} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 189 189 189 F F F 1 020-039 108 108 108 200 200 200 F F F 2 040-059 106 106 106 196 196 196 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 106 106 106 196 196 196 T T T 5 100-119 103 103 103 190 190 190 F F F 6 120-139 108 108 108 200 200 200 F F F 7 140-159 107 107 107 199 199 199 F F F 8 160-179 108 108 108 200 200 200 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 106 106 106 198 198 198 F F F 11 220-239 106 106 106 198 198 198 F F F 12 240-259 104 104 104 191 191 191 T T T 13 260-279 102 102 102 187 187 187 F F F 14 280-299 105 105 105 194 194 194 F F F 15 300-319 106 106 106 195 195 195 F F F 16 320-339 102 102 102 187 187 187 T T T 17 340-359 99 99 99 184 184 184 F F F 18 360-379 107 107 107 199 199 199 F F F 19 380-399 100 100 100 185 185 185 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 108 108 108 200 200 200 F F F 23 460-479 104 104 104 193 193 193 T T T 24 480-499 104 104 104 193 193 193 F F F 25 500-519 106 106 106 195 195 195 F F F 26 520-539 103 103 103 190 190 190 F F F 27 540-559 105 105 105 194 194 194 F F F 28 560-579 105 105 105 194 194 194 F F F 29 580-599 103 103 103 190 190 190 F F F Max gen: 108, Max flush: 200 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 189 189 189 F F F 1 020-039 108 108 108 200 200 200 F F F 2 040-059 106 106 106 196 196 196 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 109 109 109 201 201 201 F F F 5 100-119 103 103 103 190 190 190 F F F 6 120-139 108 108 108 200 200 200 F F F 7 140-159 107 107 107 199 199 199 F F F 8 160-179 108 108 108 200 200 200 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 106 106 106 198 198 198 F F F 11 220-239 106 106 106 198 198 198 F F F 12 240-259 109 109 109 201 201 201 F F F 13 260-279 102 102 102 187 187 187 F F F 14 280-299 105 105 105 194 194 194 F F F 15 300-319 106 106 106 195 195 195 F F F 16 320-339 109 109 109 201 201 201 F F F 17 340-359 99 99 99 184 184 184 F F F 18 360-379 107 107 107 199 199 199 F F F 19 380-399 100 100 100 185 185 185 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 108 108 108 200 200 200 F F F 23 460-479 109 109 109 201 201 201 F F F 24 480-499 104 104 104 193 193 193 F F F 25 500-519 106 106 106 195 195 195 F F F 26 520-539 103 103 103 190 190 190 F F F 27 540-559 105 105 105 194 194 194 F F F 28 560-579 105 105 105 194 194 194 F F F 29 580-599 103 103 103 190 190 190 F F F Max gen: 109, Max flush: 201 Begin loop 55 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.804570656Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.804911892Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.80493609Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625} {"msg":"Crucible 73f985aa-70d3-4c7e-a080-1847457c8b27 has session id: 48c4951b-1fa4-450d-8c7f-d208148a9889","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.805078036Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Upstairs opts: Upstairs UUID: 73f985aa-70d3-4c7e-a080-1847457c8b27, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.805155388Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Crucible stats registered with UUID: 73f985aa-70d3-4c7e-a080-1847457c8b27","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.805182166Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.805788335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625} {"msg":"73f985aa-70d3-4c7e-a080-1847457c8b27 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.805819932Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.80584379Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.805870907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.805894335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"1","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.805917742Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"1","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.80594207Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"2","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.805968128Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"2","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.805994575Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"io task","client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.80604593Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"io task","client":"1","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.806071647Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"io task","client":"2","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"ds_connection connected from Ok(127.0.0.1:47370)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.806096985Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"io task","client":"1","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:56.806126202Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"io task","client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:56.806151789Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"io task","client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"ds_connection connected from Ok(127.0.0.1:43965)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.806291616Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"io task","client":"2","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:44:56.806327292Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.80635303Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.806383236Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.806408264Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"io task","client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.806773358Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"2","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.806799695Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"2","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.806932602Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"1","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.807006355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"1","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.807349231Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"2","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.807486507Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"1","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.807876059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"2","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.807898546Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"2","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.807912455Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.807950271Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"1","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.80796683Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"1","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:44:56.807980558Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.80770737Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"io task","client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"ds_connection connected from Ok(127.0.0.1:44124)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.807816609Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"io task","client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.808771825Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.808806401Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.809332529Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.809878225Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.809897783Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[0]R flush_numbers[0..12]: [189, 200, 196, 198, 201, 190, 200, 199, 200, 186, 198, 198]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.809912581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[0]R generation[0..12]: [102, 108, 106, 106, 109, 103, 108, 107, 108, 101, 106, 106]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.80992677Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.809941498Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[1]R flush_numbers[0..12]: [189, 200, 196, 198, 201, 190, 200, 199, 200, 186, 198, 198]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.809955607Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[1]R generation[0..12]: [102, 108, 106, 106, 109, 103, 108, 107, 108, 101, 106, 106]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.809969636Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.809983864Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[2]R flush_numbers[0..12]: [189, 200, 196, 198, 201, 190, 200, 199, 200, 186, 198, 198]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.809997943Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[2]R generation[0..12]: [102, 108, 106, 106, 109, 103, 108, 107, 108, 101, 106, 106]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810011962Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.81002604Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810042768Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810056557Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Next flush: 202","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO time":"2024-07-09T21:45:06.810069756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810086924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"":"downstairs","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810100203Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810112352Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"0","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.81012552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"1","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810138629Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"client":"2","":"downstairs","session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810152278Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810164607Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"73f985aa-70d3-4c7e-a080-1847457c8b27 is now active with session: 48c4951b-1fa4-450d-8c7f-d208148a9889","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810177455Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810189474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625,"session_id":"48c4951b-1fa4-450d-8c7f-d208148a9889"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:06.810201773Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 329 len 24576 data: 9 10 12 11 10 10 02/30 Read block 55 len 12288 03/30 Flush 04/30 Write block 316 len 28672 data: 9 11 12 12 11 11 10 05/30 Read block 464 len 8192 06/30 Read block 209 len 36864 07/30 Flush 08/30 Write block 136 len 32768 data: 6 9 9 11 9 8 10 10 09/30 Read block 352 len 40960 10/30 Write block 343 len 8192 data: 8 8 11/30 Read block 282 len 8192 12/30 Write block 570 len 36864 data: 11 14 12 12 12 10 12 10 6 13/30 Flush 14/30 Write block 21 len 36864 data: 8 5 6 6 7 8 8 10 12 15/30 Read block 293 len 28672 16/30 Read block 1 len 8192 17/30 Flush 18/30 Write block 552 len 24576 data: 12 11 12 12 14 14 19/30 Read block 345 len 36864 20/30 Write block 58 len 8192 data: 14 12 21/30 Write block 387 len 4096 data: 13 22/30 Read block 584 len 40960 23/30 Flush 24/30 Write block 218 len 8192 data: 7 8 25/30 Read block 149 len 16384 26/30 Write block 339 len 32768 data: 7 7 8 8 9 9 8 8 27/30 Read block 301 len 32768 28/30 Read block 404 len 32768 29/30 Write block 48 len 24576 data: 7 7 4 7 6 10 30/30 Read block 569 len 4096 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 2 Done Done Done false 25 Acked 1024 Read 4 Done Done Done false 26 Acked 1025 Write 8 Done Done Done false 27 Acked 1026 Read 8 Done Done Done false 28 Acked 1027 Read 8 Done Done Done false 29 Acked 1028 Write 6 Done Done Done false 30 Acked 1029 Read 1 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:45:06.841881993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1625} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 189 189 189 F F F 1 020-039 110 110 110 205 205 205 F F F 2 040-059 110 110 110 206 206 206 T T T 3 060-079 106 106 106 198 198 198 F F F 4 080-099 109 109 109 201 201 201 F F F 5 100-119 103 103 103 190 190 190 F F F 6 120-139 110 110 110 204 204 204 F F F 7 140-159 110 110 110 204 204 204 F F F 8 160-179 108 108 108 200 200 200 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 106 106 106 198 198 198 T T T 11 220-239 106 106 106 198 198 198 F F F 12 240-259 109 109 109 201 201 201 F F F 13 260-279 102 102 102 187 187 187 F F F 14 280-299 105 105 105 194 194 194 F F F 15 300-319 110 110 110 203 203 203 F F F 16 320-339 110 110 110 203 203 203 T T T 17 340-359 110 110 110 204 204 204 T T T 18 360-379 107 107 107 199 199 199 F F F 19 380-399 110 110 110 206 206 206 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 108 108 108 200 200 200 F F F 23 460-479 109 109 109 201 201 201 F F F 24 480-499 104 104 104 193 193 193 F F F 25 500-519 106 106 106 195 195 195 F F F 26 520-539 103 103 103 190 190 190 F F F 27 540-559 110 110 110 206 206 206 F F F 28 560-579 110 110 110 204 204 204 F F F 29 580-599 103 103 103 190 190 190 F F F Max gen: 110, Max flush: 206 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 189 189 189 F F F 1 020-039 110 110 110 205 205 205 F F F 2 040-059 111 111 111 207 207 207 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 109 109 109 201 201 201 F F F 5 100-119 103 103 103 190 190 190 F F F 6 120-139 110 110 110 204 204 204 F F F 7 140-159 110 110 110 204 204 204 F F F 8 160-179 108 108 108 200 200 200 F F F 9 180-199 101 101 101 186 186 186 F F F 10 200-219 111 111 111 207 207 207 F F F 11 220-239 106 106 106 198 198 198 F F F 12 240-259 109 109 109 201 201 201 F F F 13 260-279 102 102 102 187 187 187 F F F 14 280-299 105 105 105 194 194 194 F F F 15 300-319 110 110 110 203 203 203 F F F 16 320-339 111 111 111 207 207 207 F F F 17 340-359 111 111 111 207 207 207 F F F 18 360-379 107 107 107 199 199 199 F F F 19 380-399 110 110 110 206 206 206 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 108 108 108 200 200 200 F F F 23 460-479 109 109 109 201 201 201 F F F 24 480-499 104 104 104 193 193 193 F F F 25 500-519 106 106 106 195 195 195 F F F 26 520-539 103 103 103 190 190 190 F F F 27 540-559 110 110 110 206 206 206 F F F 28 560-579 110 110 110 204 204 204 F F F 29 580-599 103 103 103 190 190 190 F F F Max gen: 111, Max flush: 207 Begin loop 56 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.055600724Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.055929892Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.055956539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633} {"msg":"Crucible a3957529-461f-47a5-8c0a-d4968f61e7d5 has session id: c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.056395465Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Upstairs opts: Upstairs UUID: a3957529-461f-47a5-8c0a-d4968f61e7d5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.056442581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Crucible stats registered with UUID: a3957529-461f-47a5-8c0a-d4968f61e7d5","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.056462749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.056796236Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633} {"msg":"a3957529-461f-47a5-8c0a-d4968f61e7d5 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.056828173Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.05685377Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.056883717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.056909354Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"1","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.056935382Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"1","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.05695988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"2","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.056986167Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"2","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.057009095Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"io task","client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.057031453Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"io task","client":"1","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.05705501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"io task","client":"2","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:45:21.057176258Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"io task","client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:45:21.057216394Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"io task","client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:45:21.057246181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.057270699Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.057365969Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"ds_connection connected from Ok(127.0.0.1:61495)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.057401026Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"io task","client":"1","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.057430163Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"io task","client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"ds_connection connected from Ok(127.0.0.1:57321)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.057455351Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"io task","client":"2","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.058048012Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"1","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.058079778Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"1","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.058106106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"2","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.058133013Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"2","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.058617145Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"2","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.058752802Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"1","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.059083679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"2","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.059105797Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"2","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.059119615Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.059196568Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"1","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.059215086Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"1","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:21.059228625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.05873856Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"io task","client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"ds_connection connected from Ok(127.0.0.1:63551)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.058894974Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"io task","client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.060120033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.06014638Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.060686467Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061202585Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061229683Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[0]R flush_numbers[0..12]: [189, 205, 207, 198, 201, 190, 204, 204, 200, 186, 207, 198]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061248351Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[0]R generation[0..12]: [102, 110, 111, 106, 109, 103, 110, 110, 108, 101, 111, 106]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061266509Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061284767Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[1]R flush_numbers[0..12]: [189, 205, 207, 198, 201, 190, 204, 204, 200, 186, 207, 198]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061302575Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[1]R generation[0..12]: [102, 110, 111, 106, 109, 103, 110, 110, 108, 101, 111, 106]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061319804Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061337602Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[2]R flush_numbers[0..12]: [189, 205, 207, 198, 201, 190, 204, 204, 200, 186, 207, 198]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.06135503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[2]R generation[0..12]: [102, 110, 111, 106, 109, 103, 110, 110, 108, 101, 111, 106]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:45:31.061372369Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061396686Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Max found gen is 112","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061413524Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Generation requested: 112 >= found:112","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061431163Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Next flush: 208","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061447571Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.06146146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"":"downstairs","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061474829Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061486867Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"0","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061508515Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"1","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061523174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"client":"2","":"downstairs","session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061536742Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061548901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"a3957529-461f-47a5-8c0a-d4968f61e7d5 is now active with session: c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061566029Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061588047Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633,"session_id":"c8c979b5-3ca7-4b62-80a1-7a3e1b8f6aa2"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:31.061604925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 114 len 28672 data: 13 12 10 10 6 6 5 03/30 Flush 04/30 Flush 05/30 Write block 224 len 8192 data: 9 8 06/30 Read block 401 len 16384 07/30 Write block 263 len 28672 data: 10 9 11 10 9 4 6 08/30 Flush 09/30 Write block 206 len 20480 data: 13 15 15 13 12 10/30 Write block 445 len 32768 data: 12 11 12 12 11 10 11 9 11/30 Write block 381 len 24576 data: 6 4 6 7 11 13 12/30 Read block 454 len 40960 13/30 Write block 217 len 36864 data: 11 8 9 7 10 12 10 10 9 14/30 Write block 449 len 28672 data: 12 11 12 10 9 10 10 15/30 Write block 6 len 28672 data: 4 6 7 8 9 9 11 16/30 Write block 281 len 24576 data: 7 6 7 8 9 6 17/30 Read block 486 len 12288 18/30 Read block 39 len 16384 19/30 Read block 213 len 36864 20/30 Read block 400 len 8192 21/30 Read block 136 len 8192 22/30 Write block 36 len 36864 data: 13 16 10 11 9 7 7 9 9 23/30 Write block 153 len 8192 data: 7 6 24/30 Write block 484 len 20480 data: 7 8 9 11 11 25/30 Write block 381 len 40960 data: 7 5 7 8 12 14 14 15 17 16 26/30 Write block 590 len 24576 data: 13 13 10 11 13 10 27/30 Read block 40 len 40960 28/30 Write block 448 len 32768 data: 13 13 12 13 11 10 11 11 29/30 Write block 520 len 12288 data: 7 5 5 30/30 Write block 196 len 20480 data: 7 9 10 10 10 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Read 4 Done Done Done false 20 Acked 1019 Read 9 Done Done Done false 21 Acked 1020 Read 2 Done Done Done false 22 Acked 1021 Read 2 Done Done Done false 23 Acked 1022 Write 9 Done Done Done false 24 Acked 1023 Write 2 Done Done Done false 25 Acked 1024 Write 5 Done Done Done false 26 Acked 1025 Write 10 Done Done Done false 27 Acked 1026 Write 6 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Write 3 Done Done Done false 31 Acked 1030 Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 12 12 12 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:45:31.09041879Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1633} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 112 112 212 212 212 F F F 1 020-039 110 110 110 205 205 205 T T T 2 040-059 111 111 111 207 207 207 T T T 3 060-079 106 106 106 198 198 198 F F F 4 080-099 109 109 109 201 201 201 F F F 5 100-119 112 112 112 209 209 209 F F F 6 120-139 112 112 112 209 209 209 F F F 7 140-159 110 110 110 204 204 204 T T T 8 160-179 108 108 108 200 200 200 F F F 9 180-199 101 101 101 186 186 186 T T T 10 200-219 112 112 112 212 212 212 T T T 11 220-239 112 112 112 212 212 212 F F F 12 240-259 109 109 109 201 201 201 F F F 13 260-279 112 112 112 211 211 211 F F F 14 280-299 112 112 112 212 212 212 F F F 15 300-319 110 110 110 203 203 203 F F F 16 320-339 111 111 111 207 207 207 F F F 17 340-359 111 111 111 207 207 207 F F F 18 360-379 107 107 107 199 199 199 F F F 19 380-399 112 112 112 212 212 212 T T T 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 112 112 112 212 212 212 T T T 23 460-479 109 109 109 201 201 201 F F F 24 480-499 104 104 104 193 193 193 T T T 25 500-519 106 106 106 195 195 195 F F F 26 520-539 103 103 103 190 190 190 T T T 27 540-559 110 110 110 206 206 206 F F F 28 560-579 110 110 110 204 204 204 F F F 29 580-599 103 103 103 190 190 190 T T T Max gen: 112, Max flush: 212 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 112 112 212 212 212 F F F 1 020-039 113 113 113 213 213 213 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 109 109 109 201 201 201 F F F 5 100-119 112 112 112 209 209 209 F F F 6 120-139 112 112 112 209 209 209 F F F 7 140-159 113 113 113 213 213 213 F F F 8 160-179 108 108 108 200 200 200 F F F 9 180-199 113 113 113 213 213 213 F F F 10 200-219 113 113 113 213 213 213 F F F 11 220-239 112 112 112 212 212 212 F F F 12 240-259 109 109 109 201 201 201 F F F 13 260-279 112 112 112 211 211 211 F F F 14 280-299 112 112 112 212 212 212 F F F 15 300-319 110 110 110 203 203 203 F F F 16 320-339 111 111 111 207 207 207 F F F 17 340-359 111 111 111 207 207 207 F F F 18 360-379 107 107 107 199 199 199 F F F 19 380-399 113 113 113 213 213 213 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 113 113 113 213 213 213 F F F 23 460-479 109 109 109 201 201 201 F F F 24 480-499 113 113 113 213 213 213 F F F 25 500-519 106 106 106 195 195 195 F F F 26 520-539 113 113 113 213 213 213 F F F 27 540-559 110 110 110 206 206 206 F F F 28 560-579 110 110 110 204 204 204 F F F 29 580-599 113 113 113 213 213 213 F F F Max gen: 113, Max flush: 213 Begin loop 57 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.391475544Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.391846857Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.391874594Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641} {"msg":"Crucible 1a5d9857-ccae-43be-ab7f-79161bb77aaa has session id: 0d65e5fb-a508-4e82-8e65-37a2f3229cb4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392262406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Upstairs opts: Upstairs UUID: 1a5d9857-ccae-43be-ab7f-79161bb77aaa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392361406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Crucible stats registered with UUID: 1a5d9857-ccae-43be-ab7f-79161bb77aaa","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392383254Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392658077Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641} {"msg":"1a5d9857-ccae-43be-ab7f-79161bb77aaa active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392685794Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392699353Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392713771Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.3927277Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"1","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392741449Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"1","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392755057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"2","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392768576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"2","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.39282113Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"io task","client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392858237Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"io task","client":"1","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.392885354Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"io task","client":"2","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"ds_connection connected from Ok(127.0.0.1:62403)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.393037739Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"io task","client":"2","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:45:45.393079075Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"io task","client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"ds_connection connected from Ok(127.0.0.1:58949)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.393108302Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"io task","client":"1","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:45:45.393138219Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"io task","client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:45:45.393169666Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.393196103Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.393223431Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.393247688Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"io task","client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.393762707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"1","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.393789205Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"1","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.393803763Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"2","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.393825191Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"2","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.394459758Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"2","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.394495895Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"1","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.395030611Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"1","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.395056449Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"1","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.395071458Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.395084036Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"2","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.395097785Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"2","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:45.395115283Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.393914808Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"io task","client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"ds_connection connected from Ok(127.0.0.1:44119)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.394134896Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"io task","client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.395150006Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.395197581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.395772894Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396392362Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.39642144Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[0]R flush_numbers[0..12]: [212, 213, 213, 198, 201, 209, 209, 213, 200, 213, 213, 212]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396441507Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[0]R generation[0..12]: [112, 113, 113, 106, 109, 112, 112, 113, 108, 113, 113, 112]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396461136Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396479174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[1]R flush_numbers[0..12]: [212, 213, 213, 198, 201, 209, 209, 213, 200, 213, 213, 212]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396496452Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[1]R generation[0..12]: [112, 113, 113, 106, 109, 112, 112, 113, 108, 113, 113, 112]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.39651508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396533278Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[2]R flush_numbers[0..12]: [212, 213, 213, 198, 201, 209, 209, 213, 200, 213, 213, 212]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396556636Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} Wait for a query_work_queue command to finish before sending IO {"msg":"[2]R generation[0..12]: [112, 113, 113, 106, 109, 112, 112, 113, 108, 113, 113, 112]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396578154Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396596632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.39661309Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396630899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Next flush: 214","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396648277Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396664476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"":"downstairs","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396681924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396696962Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"0","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396714431Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"1","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396731119Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"client":"2","":"downstairs","session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396748677Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396764335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"1a5d9857-ccae-43be-ab7f-79161bb77aaa is now active with session: 0d65e5fb-a508-4e82-8e65-37a2f3229cb4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396780654Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396795692Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641,"session_id":"0d65e5fb-a508-4e82-8e65-37a2f3229cb4"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:45:55.396811071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 366 len 16384 data: 9 8 9 12 03/30 Read block 497 len 28672 04/30 Read block 160 len 20480 05/30 Read block 116 len 20480 06/30 Read block 420 len 8192 07/30 Read block 343 len 20480 08/30 Flush 09/30 Write block 191 len 20480 data: 12 11 12 12 10 10/30 Read block 134 len 20480 11/30 Write block 236 len 32768 data: 9 8 9 5 7 6 6 6 12/30 Read block 65 len 16384 13/30 Read block 509 len 24576 14/30 Read block 207 len 24576 15/30 Write block 372 len 16384 data: 7 8 10 11 16/30 Read block 125 len 36864 17/30 Read block 222 len 40960 18/30 Write block 277 len 32768 data: 6 6 6 7 8 7 8 9 19/30 Read block 302 len 28672 20/30 Read block 109 len 40960 21/30 Write block 171 len 20480 data: 5 4 3 3 7 22/30 Read block 88 len 20480 23/30 Read block 420 len 24576 24/30 Read block 73 len 32768 25/30 Write block 318 len 40960 data: 13 13 12 12 11 10 10 11 10 10 26/30 Read block 458 len 4096 27/30 Read block 200 len 4096 28/30 Read block 428 len 24576 29/30 Write block 310 len 36864 data: 10 9 11 11 12 9 10 12 14 30/30 Write block 128 len 16384 data: 6 6 5 6 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 5 Done Done Done false 23 Acked 1022 Read 5 Done Done Done false 24 Acked 1023 Read 6 Done Done Done false 25 Acked 1024 Read 8 Done Done Done false 26 Acked 1025 Write 10 Done Done Done false 27 Acked 1026 Read 1 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Read 6 Done Done Done false 30 Acked 1029 Write 9 Done Done Done false 31 Acked 1030 Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 9 9 9 27 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:45:55.423922354Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1641} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 112 112 212 212 212 F F F 1 020-039 113 113 113 213 213 213 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 109 109 109 201 201 201 F F F 5 100-119 112 112 112 209 209 209 F F F 6 120-139 112 112 112 209 209 209 T T T 7 140-159 113 113 113 213 213 213 F F F 8 160-179 108 108 108 200 200 200 T T T 9 180-199 114 114 114 216 216 216 F F F 10 200-219 113 113 113 213 213 213 F F F 11 220-239 114 114 114 216 216 216 F F F 12 240-259 114 114 114 216 216 216 F F F 13 260-279 114 114 114 216 216 216 F F F 14 280-299 114 114 114 216 216 216 F F F 15 300-319 110 110 110 203 203 203 T T T 16 320-339 111 111 111 207 207 207 T T T 17 340-359 111 111 111 207 207 207 F F F 18 360-379 114 114 114 216 216 216 F F F 19 380-399 113 113 113 213 213 213 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 113 113 113 213 213 213 F F F 23 460-479 109 109 109 201 201 201 F F F 24 480-499 113 113 113 213 213 213 F F F 25 500-519 106 106 106 195 195 195 F F F 26 520-539 113 113 113 213 213 213 F F F 27 540-559 110 110 110 206 206 206 F F F 28 560-579 110 110 110 204 204 204 F F F 29 580-599 113 113 113 213 213 213 F F F Max gen: 114, Max flush: 216 A Difference in extent metadata is expected here On loop 57 Verifying data now No change after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 112 112 212 212 212 F F F 1 020-039 113 113 113 213 213 213 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 109 109 109 201 201 201 F F F 5 100-119 112 112 112 209 209 209 F F F 6 120-139 115 115 115 217 217 217 F F F 7 140-159 113 113 113 213 213 213 F F F 8 160-179 115 115 115 217 217 217 F F F 9 180-199 114 114 114 216 216 216 F F F 10 200-219 113 113 113 213 213 213 F F F 11 220-239 114 114 114 216 216 216 F F F 12 240-259 114 114 114 216 216 216 F F F 13 260-279 114 114 114 216 216 216 F F F 14 280-299 114 114 114 216 216 216 F F F 15 300-319 115 115 115 217 217 217 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 111 111 111 207 207 207 F F F 18 360-379 114 114 114 216 216 216 F F F 19 380-399 113 113 113 213 213 213 F F F 20 400-419 104 104 104 192 192 192 F F F 21 420-439 105 105 105 194 194 194 F F F 22 440-459 113 113 113 213 213 213 F F F 23 460-479 109 109 109 201 201 201 F F F 24 480-499 113 113 113 213 213 213 F F F 25 500-519 106 106 106 195 195 195 F F F 26 520-539 113 113 113 213 213 213 F F F 27 540-559 110 110 110 206 206 206 F F F 28 560-579 110 110 110 204 204 204 F F F 29 580-599 113 113 113 213 213 213 F F F Max gen: 115, Max flush: 217 Begin loop 58 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.632973729Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.633313635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.633338143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649} {"msg":"Crucible 5b964df8-8975-4388-8313-f25c940199d6 has session id: 2f22d35a-65fa-47f1-9c27-4170bfe43f25","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.6337691Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Upstairs opts: Upstairs UUID: 5b964df8-8975-4388-8313-f25c940199d6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.633811886Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Crucible stats registered with UUID: 5b964df8-8975-4388-8313-f25c940199d6","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.633839583Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634138243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649} {"msg":"5b964df8-8975-4388-8313-f25c940199d6 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634243893Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634262631Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.63427774Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634291698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"1","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634305547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"1","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634318745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"2","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634332314Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"2","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634396548Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"io task","client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634415746Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"io task","client":"2","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634429954Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"io task","client":"1","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:09.634761782Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"io task","client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"ds_connection connected from Ok(127.0.0.1:50539)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634801378Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"io task","client":"1","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:09.634829325Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"io task","client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:09.634849923Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634863071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.63487671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"ds_connection connected from Ok(127.0.0.1:51235)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634890349Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"io task","client":"2","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.634911857Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"io task","client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.635458183Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"2","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.63548525Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"2","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.635500918Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"1","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.635514847Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"1","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.636060303Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"1","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.63618987Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"2","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.636529117Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"1","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.636566013Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"1","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.63659441Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.636620827Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"2","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.636652714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"2","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:09.636674082Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.635468997Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"io task","client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"ds_connection connected from Ok(127.0.0.1:43861)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.635762997Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"io task","client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.636875407Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.636909664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.637527002Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638087307Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638110915Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[0]R flush_numbers[0..12]: [212, 213, 213, 198, 201, 209, 217, 213, 217, 216, 213, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638126123Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[0]R generation[0..12]: [112, 113, 113, 106, 109, 112, 115, 113, 115, 114, 113, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638140472Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.63815503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[1]R flush_numbers[0..12]: [212, 213, 213, 198, 201, 209, 217, 213, 217, 216, 213, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638169309Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[1]R generation[0..12]: [112, 113, 113, 106, 109, 112, 115, 113, 115, 114, 113, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638183337Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638197576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[2]R flush_numbers[0..12]: [212, 213, 213, 198, 201, 209, 217, 213, 217, 216, 213, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638215364Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[2]R generation[0..12]: [112, 113, 113, 106, 109, 112, 115, 113, 115, 114, 113, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638229863Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638244051Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Max found gen is 116","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.63826067Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Generation requested: 116 >= found:116","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638274488Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Next flush: 218","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638287597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638300656Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"":"downstairs","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638313924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638328093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"0","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638351461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"1","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638374689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"client":"2","":"downstairs","session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638398826Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638420584Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"5b964df8-8975-4388-8313-f25c940199d6 is now active with session: 2f22d35a-65fa-47f1-9c27-4170bfe43f25","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638444152Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638467989Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649,"session_id":"2f22d35a-65fa-47f1-9c27-4170bfe43f25"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:19.638495277Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 417 len 16384 data: 7 7 7 9 02/30 Read block 326 len 40960 03/30 Read block 358 len 36864 04/30 Read block 70 len 32768 05/30 Write block 30 len 36864 data: 11 11 13 12 11 13 14 17 11 06/30 Read block 402 len 8192 07/30 Flush 08/30 Write block 129 len 12288 data: 7 6 7 09/30 Flush 10/30 Write block 165 len 20480 data: 11 10 9 7 5 11/30 Flush 12/30 Write block 101 len 24576 data: 10 9 11 10 9 9 13/30 Write block 140 len 8192 data: 10 9 14/30 Write block 84 len 28672 data: 10 9 12 12 12 10 9 15/30 Read block 53 len 20480 16/30 Write block 185 len 20480 data: 6 6 8 7 8 17/30 Write block 81 len 20480 data: 8 10 10 11 10 18/30 Read block 211 len 36864 19/30 Read block 381 len 28672 20/30 Read block 280 len 20480 21/30 Flush 22/30 Write block 571 len 40960 data: 15 13 13 13 11 13 11 7 6 6 23/30 Read block 500 len 16384 24/30 Flush 25/30 Write block 506 len 36864 data: 10 9 11 9 11 11 12 9 9 26/30 Read block 335 len 4096 27/30 Read block 523 len 28672 28/30 Write block 100 len 36864 data: 11 11 10 12 11 10 10 9 11 29/30 Read block 307 len 32768 30/30 Read block 224 len 36864 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Write 9 Done Done Done false 27 Acked 1026 Read 1 Done Done Done false 28 Acked 1027 Read 7 Done Done Done false 29 Acked 1028 Write 9 Done Done Done false 30 Acked 1029 Read 8 Done Done Done false 31 Acked 1030 Read 9 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 0 2 Done 5 5 6 16 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:19.670577377Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1649} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 112 112 212 212 212 F F F 1 020-039 116 116 116 218 218 218 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 116 116 116 222 222 222 F F F 5 100-119 116 116 116 221 221 221 T T T 6 120-139 116 116 116 219 219 219 F F F 7 140-159 116 116 116 221 221 221 F F F 8 160-179 116 116 116 220 220 220 F F F 9 180-199 116 116 116 221 221 221 F F F 10 200-219 113 113 113 213 213 213 F F F 11 220-239 114 114 114 216 216 216 F F F 12 240-259 114 114 114 216 216 216 F F F 13 260-279 114 114 114 216 216 216 F F F 14 280-299 114 114 114 216 216 216 F F F 15 300-319 115 115 115 217 217 217 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 111 111 111 207 207 207 F F F 18 360-379 114 114 114 216 216 216 F F F 19 380-399 113 113 113 213 213 213 F F F 20 400-419 116 116 116 218 218 218 F F F 21 420-439 116 116 116 218 218 218 F F F 22 440-459 113 113 113 213 213 213 F F F 23 460-479 109 109 109 201 201 201 F F F 24 480-499 113 113 113 213 213 213 F F F 25 500-519 106 106 106 195 195 195 T T T 26 520-539 113 113 113 213 213 213 F F F 27 540-559 110 110 110 206 206 206 F F F 28 560-579 116 116 116 223 223 223 F F F 29 580-599 116 116 116 223 223 223 F F F Max gen: 116, Max flush: 223 A Difference in extent metadata is expected here On loop 58 Verifying data now No change after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 112 112 212 212 212 F F F 1 020-039 116 116 116 218 218 218 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 116 116 116 222 222 222 F F F 5 100-119 117 117 117 224 224 224 F F F 6 120-139 116 116 116 219 219 219 F F F 7 140-159 116 116 116 221 221 221 F F F 8 160-179 116 116 116 220 220 220 F F F 9 180-199 116 116 116 221 221 221 F F F 10 200-219 113 113 113 213 213 213 F F F 11 220-239 114 114 114 216 216 216 F F F 12 240-259 114 114 114 216 216 216 F F F 13 260-279 114 114 114 216 216 216 F F F 14 280-299 114 114 114 216 216 216 F F F 15 300-319 115 115 115 217 217 217 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 111 111 111 207 207 207 F F F 18 360-379 114 114 114 216 216 216 F F F 19 380-399 113 113 113 213 213 213 F F F 20 400-419 116 116 116 218 218 218 F F F 21 420-439 116 116 116 218 218 218 F F F 22 440-459 113 113 113 213 213 213 F F F 23 460-479 109 109 109 201 201 201 F F F 24 480-499 113 113 113 213 213 213 F F F 25 500-519 117 117 117 224 224 224 F F F 26 520-539 113 113 113 213 213 213 F F F 27 540-559 110 110 110 206 206 206 F F F 28 560-579 116 116 116 223 223 223 F F F 29 580-599 116 116 116 223 223 223 F F F Max gen: 117, Max flush: 224 Begin loop 59 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.882193576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.882599786Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.882630443Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657} {"msg":"Crucible 7c30aa3a-9881-4de7-aa6a-a33ce43032fa has session id: 9634e8da-c838-4f5d-a658-ddb5438a7711","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.882903096Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Upstairs opts: Upstairs UUID: 7c30aa3a-9881-4de7-aa6a-a33ce43032fa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.88296171Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Crucible stats registered with UUID: 7c30aa3a-9881-4de7-aa6a-a33ce43032fa","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.882993867Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883320805Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657} {"msg":"7c30aa3a-9881-4de7-aa6a-a33ce43032fa active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883393897Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883412156Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883434313Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883459981Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"1","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883484578Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"1","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883506996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"2","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883530744Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"2","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883555881Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"io task","client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883579249Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"io task","client":"1","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883602827Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"io task","client":"2","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"ds_connection connected from Ok(127.0.0.1:55099)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883742513Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"io task","client":"1","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"ds_connection connected from Ok(127.0.0.1:53705)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.883778119Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"io task","client":"2","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:33.884062851Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"io task","client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:33.884106387Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"io task","client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:33.884137654Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.884161501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.884185389Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.884300377Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"io task","client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.884605157Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"1","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.884642274Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"1","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.884671081Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"2","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.884697348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"2","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.885271021Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"2","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.885309597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"1","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.885814678Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"2","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.885850394Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"2","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.885876891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.885934515Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"1","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.885965332Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"1","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:33.885982091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.885715094Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"io task","client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"ds_connection connected from Ok(127.0.0.1:36276)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.885839592Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"io task","client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.887012595Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.887038323Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.887609336Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888165301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888192858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[0]R flush_numbers[0..12]: [212, 218, 213, 198, 222, 224, 219, 221, 220, 221, 213, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888212106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[0]R generation[0..12]: [112, 116, 113, 106, 116, 117, 116, 116, 116, 116, 113, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888229835Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888247733Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[1]R flush_numbers[0..12]: [212, 218, 213, 198, 222, 224, 219, 221, 220, 221, 213, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888265271Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[1]R generation[0..12]: [112, 116, 113, 106, 116, 117, 116, 116, 116, 116, 113, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888283409Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888301037Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[2]R flush_numbers[0..12]: [212, 218, 213, 198, 222, 224, 219, 221, 220, 221, 213, 216]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:46:43.888324045Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[2]R generation[0..12]: [112, 116, 113, 106, 116, 117, 116, 116, 116, 116, 113, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888345473Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888364691Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Max found gen is 118","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.88838132Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Generation requested: 118 >= found:118","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888398858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Next flush: 225","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888425755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888452053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"":"downstairs","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888471031Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888486199Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"0","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888502868Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"1","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888519396Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"client":"2","":"downstairs","session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888537244Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888552843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"7c30aa3a-9881-4de7-aa6a-a33ce43032fa is now active with session: 9634e8da-c838-4f5d-a658-ddb5438a7711","v":0,"name":"crucible","level":30Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2024-07-09T21:46:43.888569041Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888590569Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657,"session_id":"9634e8da-c838-4f5d-a658-ddb5438a7711"} {"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:43.888606337Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 568 len 36864 data: 11 10 12 16 14 14 14 12 14 02/30 Write block 369 len 32768 data: 13 10 7 8 9 11 12 10 03/30 Write block 214 len 24576 data: 11 11 10 12 9 10 04/30 Read block 358 len 8192 05/30 Read block 360 len 20480 06/30 Read block 170 len 40960 07/30 Flush 08/30 Write block 19 len 28672 data: 6 6 9 6 7 7 8 09/30 Write block 413 len 40960 data: 8 8 9 9 8 8 8 10 9 9 10/30 Flush 11/30 Write block 278 len 20480 data: 7 7 8 9 8 12/30 Write block 482 len 4096 data: 5 13/30 Read block 392 len 36864 14/30 Read block 338 len 12288 15/30 Write block 593 len 4096 data: 12 16/30 Write block 434 len 36864 data: 13 12 9 9 11 8 10 12 12 17/30 Read block 83 len 4096 18/30 Read block 250 len 16384 19/30 Write block 516 len 8192 data: 6 9 20/30 Read block 393 len 40960 21/30 Write block 358 len 8192 data: 13 11 22/30 Read block 236 len 28672 23/30 Write block 512 len 36864 data: 13 10 10 8 7 10 8 7 8 24/30 Write block 103 len 4096 data: 13 25/30 Write block 527 len 32768 data: 8 8 7 5 5 5 6 5 26/30 Read block 340 len 20480 27/30 Write block 369 len 36864 data: 14 11 8 9 10 12 13 11 6 28/30 Write block 149 len 16384 data: 6 6 7 7 29/30 Read block 562 len 32768 30/30 Write block 592 len 24576 data: 11 13 14 11 9 7 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:17 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 15 Acked 1014 Read 3 Done Done Done false 16 Acked 1015 Write 1 Done Done Done false 17 Acked 1016 Write 9 Done Done Done false 18 Acked 1017 Read 1 Done Done Done false 19 Acked 1018 Read 4 Done Done Done false 20 Acked 1019 Write 2 Done Done Done false 21 Acked 1020 Read 10 Done Done Done false 22 Acked 1021 Write 2 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Write 9 Done Done Done false 25 Acked 1024 Write 1 Done Done Done false 26 Acked 1025 Write 8 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Write 4 Done Done Done false 30 Acked 1029 Read 8 Done Done Done false 31 Acked 1030 Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 16 16 16 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1013 1013 1013 Downstairs last five completed: 1013 1012 1011 1010 1009 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:43.916409062Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1657} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 226 226 226 F F F 1 020-039 118 118 118 226 226 226 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 116 116 116 222 222 222 F F F 5 100-119 117 117 117 224 224 224 T T T 6 120-139 116 116 116 219 219 219 F F F 7 140-159 116 116 116 221 221 221 T T T 8 160-179 116 116 116 220 220 220 F F F 9 180-199 116 116 116 221 221 221 F F F 10 200-219 118 118 118 225 225 225 F F F 11 220-239 114 114 114 216 216 216 F F F 12 240-259 114 114 114 216 216 216 F F F 13 260-279 118 118 118 227 227 227 F F F 14 280-299 118 118 118 227 227 227 F F F 15 300-319 115 115 115 217 217 217 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 111 111 111 207 207 207 T T T 18 360-379 118 118 118 225 225 225 T T T 19 380-399 113 113 113 213 213 213 F F F 20 400-419 118 118 118 226 226 226 F F F 21 420-439 118 118 118 226 226 226 T T T 22 440-459 113 113 113 213 213 213 T T T 23 460-479 109 109 109 201 201 201 F F F 24 480-499 118 118 118 227 227 227 F F F 25 500-519 117 117 117 224 224 224 T T T 26 520-539 113 113 113 213 213 213 T T T 27 540-559 110 110 110 206 206 206 F F F 28 560-579 118 118 118 225 225 225 F F F 29 580-599 116 116 116 223 223 223 T T T Max gen: 118, Max flush: 227 A Difference in extent metadata is expected here On loop 59 Verifying data now No change after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 226 226 226 F F F 1 020-039 118 118 118 226 226 226 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 106 106 106 198 198 198 F F F 4 080-099 116 116 116 222 222 222 F F F 5 100-119 119 119 119 228 228 228 F F F 6 120-139 116 116 116 219 219 219 F F F 7 140-159 119 119 119 228 228 228 F F F 8 160-179 116 116 116 220 220 220 F F F 9 180-199 116 116 116 221 221 221 F F F 10 200-219 118 118 118 225 225 225 F F F 11 220-239 114 114 114 216 216 216 F F F 12 240-259 114 114 114 216 216 216 F F F 13 260-279 118 118 118 227 227 227 F F F 14 280-299 118 118 118 227 227 227 F F F 15 300-319 115 115 115 217 217 217 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 119 119 119 228 228 228 F F F 18 360-379 119 119 119 228 228 228 F F F 19 380-399 113 113 113 213 213 213 F F F 20 400-419 118 118 118 226 226 226 F F F 21 420-439 119 119 119 228 228 228 F F F 22 440-459 119 119 119 228 228 228 F F F 23 460-479 109 109 109 201 201 201 F F F 24 480-499 118 118 118 227 227 227 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 119 119 119 228 228 228 F F F 27 540-559 110 110 110 206 206 206 F F F 28 560-579 118 118 118 225 225 225 F F F 29 580-599 119 119 119 228 228 228 F F F Max gen: 119, Max flush: 228 Begin loop 60 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.197939883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.198418636Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.198456422Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665} {"msg":"Crucible beb72ede-8bc4-49fa-84d7-0a60f752c21b has session id: c7ea7345-dff0-4949-9339-cd5ce8b44188","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.198723125Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Upstairs opts: Upstairs UUID: beb72ede-8bc4-49fa-84d7-0a60f752c21b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.198812397Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Crucible stats registered with UUID: beb72ede-8bc4-49fa-84d7-0a60f752c21b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.198848833Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199177671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665} {"msg":"beb72ede-8bc4-49fa-84d7-0a60f752c21b active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199204368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199227815Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"0","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199253633Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"0","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199276601Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"1","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199299708Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"1","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199324526Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199347654Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199370101Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"io task","client":"0","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199391599Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"io task","client":"1","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199406488Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"io task","client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"ds_connection connected from Ok(127.0.0.1:38353)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199607028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"io task","client":"1","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"ds_connection connected from Ok(127.0.0.1:61532)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199643914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"io task","client":"0","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:58.199752894Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"io task","client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:58.19979222Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"io task","client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:46:58.199813897Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199827086Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199840595Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.199954994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"io task","client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.200320707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"0","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.200346385Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"0","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.200360633Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"1","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.200384091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"1","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.200839526Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"1","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.200947675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"0","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.201307759Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"1","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.201333507Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"1","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.201348125Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.201388391Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"0","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.20140551Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"0","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:46:58.201418958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.200873557Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"io task","client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"ds_connection connected from Ok(127.0.0.1:55221)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.20103999Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"io task","client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.202377818Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.202424433Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.202978098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203500736Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203523584Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[0]R flush_numbers[0..12]: [226, 226, 213, 198, 222, 228, 219, 228, 220, 221, 225, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203538873Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[0]R generation[0..12]: [118, 118, 113, 106, 116, 119, 116, 119, 116, 116, 118, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203553611Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.20356915Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[1]R flush_numbers[0..12]: [226, 226, 213, 198, 222, 228, 219, 228, 220, 221, 225, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203587658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[1]R generation[0..12]: [118, 118, 113, 106, 116, 119, 116, 119, 116, 116, 118, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203601966Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203616275Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[2]R flush_numbers[0..12]: [226, 226, 213, 198, 222, 228, 219, 228, 220, 221, 225, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203630124Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[2]R generation[0..12]: [118, 118, 113, 106, 116, 119, 116, 119, 116, 116, 118, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203643902Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203658161Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Max found gen is 120"Wait for a query_work_queue command to finish before sending IO ,"v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203674699Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203688078Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Next flush: 229","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203701046Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203714165Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"":"downstairs","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203727324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203747032Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"0","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.2037671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"1","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203787248Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"client":"2","":"downstairs","session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203815055Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203839523Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"beb72ede-8bc4-49fa-84d7-0a60f752c21b is now active with session: c7ea7345-dff0-4949-9339-cd5ce8b44188","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.20386427Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203878679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665,"session_id":"c7ea7345-dff0-4949-9339-cd5ce8b44188"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:08.203891168Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 469 len 8192 data: 10 9 02/30 Write block 595 len 12288 data: 12 10 8 03/30 Flush 04/30 Flush 05/30 Flush 06/30 Write block 485 len 12288 data: 9 10 12 07/30 Read block 513 len 40960 08/30 Read block 151 len 40960 09/30 Write block 20 len 12288 data: 7 10 7 10/30 Read block 203 len 32768 11/30 Read block 505 len 28672 12/30 Read block 121 len 36864 13/30 Write block 76 len 12288 data: 8 8 8 14/30 Write block 479 len 32768 data: 6 6 5 6 7 8 10 11 15/30 Read block 194 len 12288 16/30 Write block 65 len 40960 data: 9 10 8 9 10 11 12 11 10 10 17/30 Write block 526 len 36864 data: 10 9 9 8 6 6 6 7 6 18/30 Read block 498 len 24576 19/30 Write block 171 len 20480 data: 6 5 4 4 8 20/30 Write block 524 len 32768 data: 8 7 11 10 10 9 7 7 21/30 Read block 237 len 16384 22/30 Write block 65 len 28672 data: 10 11 9 10 11 12 13 23/30 Flush 24/30 Write block 106 len 4096 data: 11 25/30 Write block 317 len 8192 data: 13 15 26/30 Read block 173 len 20480 27/30 Write block 122 len 28672 data: 7 11 13 13 13 9 7 28/30 Read block 355 len 12288 29/30 Write block 373 len 8192 data: 11 13 30/30 Write block 217 len 4096 data: 13 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 1 Done Done Done false 26 Acked 1025 Write 2 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Read 3 Done Done Done false 30 Acked 1029 Write 2 Sent Sent Sent false 31 Acked 1030 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 2 2 6 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:47:08.230349905Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1665} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 226 226 226 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 120 120 120 233 233 233 F F F 4 080-099 116 116 116 222 222 222 F F F 5 100-119 119 119 119 228 228 228 T T T 6 120-139 116 116 116 219 219 219 T T T 7 140-159 119 119 119 228 228 228 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 116 116 116 221 221 221 F F F 10 200-219 118 118 118 225 225 225 T T T 11 220-239 114 114 114 216 216 216 F F F 12 240-259 114 114 114 216 216 216 F F F 13 260-279 118 118 118 227 227 227 F F F 14 280-299 118 118 118 227 227 227 F F F 15 300-319 115 115 115 217 217 217 T T T 16 320-339 115 115 115 217 217 217 F F F 17 340-359 119 119 119 228 228 228 F F F 18 360-379 119 119 119 228 228 228 T T T 19 380-399 113 113 113 213 213 213 F F F 20 400-419 118 118 118 226 226 226 F F F 21 420-439 119 119 119 228 228 228 F F F 22 440-459 119 119 119 228 228 228 F F F 23 460-479 120 120 120 232 232 232 F F F 24 480-499 120 120 120 232 232 232 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 120 120 120 233 233 233 F F F 27 540-559 110 110 110 206 206 206 F F F 28 560-579 118 118 118 225 225 225 F F F 29 580-599 120 120 120 229 229 229 F F F Max gen: 120, Max flush: 233 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 226 226 226 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 120 120 120 233 233 233 F F F 4 080-099 116 116 116 222 222 222 F F F 5 100-119 121 121 121 234 234 234 F F F 6 120-139 121 121 121 234 234 234 F F F 7 140-159 119 119 119 228 228 228 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 116 116 116 221 221 221 F F F 10 200-219 121 121 121 234 234 234 F F F 11 220-239 114 114 114 216 216 216 F F F 12 240-259 114 114 114 216 216 216 F F F 13 260-279 118 118 118 227 227 227 F F F 14 280-299 118 118 118 227 227 227 F F F 15 300-319 121 121 121 234 234 234 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 119 119 119 228 228 228 F F F 18 360-379 121 121 121 234 234 234 F F F 19 380-399 113 113 113 213 213 213 F F F 20 400-419 118 118 118 226 226 226 F F F 21 420-439 119 119 119 228 228 228 F F F 22 440-459 119 119 119 228 228 228 F F F 23 460-479 120 120 120 232 232 232 F F F 24 480-499 120 120 120 232 232 232 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 120 120 120 233 233 233 F F F 27 540-559 110 110 110 206 206 206 F F F 28 560-579 118 118 118 225 225 225 F F F 29 580-599 120 120 120 229 229 229 F F F Max gen: 121, Max flush: 234 Begin loop 61 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.457606755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.457971298Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.457999376Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673} {"msg":"Crucible 3ccb7a58-9c59-41e5-b18c-b918a008ad12 has session id: 61968bbf-8a66-47e7-a4c1-56330ea966d5","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.45836361Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Upstairs opts: Upstairs UUID: 3ccb7a58-9c59-41e5-b18c-b918a008ad12, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.45845863Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Crucible stats registered with UUID: 3ccb7a58-9c59-41e5-b18c-b918a008ad12","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.458488717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.458770019Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673} {"msg":"3ccb7a58-9c59-41e5-b18c-b918a008ad12 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.45886308Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.458885748Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"0","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.458900966Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"0","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.458914965Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.458928704Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.458942072Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"2","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.458955631Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"2","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.45896888Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"io task","client":"0","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.458982838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"io task","client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.458996627Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"io task","client":"2","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"ds_connection connected from Ok(127.0.0.1:42170)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.459155781Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"io task","client":"0","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"ds_connection connected from Ok(127.0.0.1:49497)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.459334833Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"io task","client":"2","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:47:22.45937118Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"io task","client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:47:22.459389938Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"io task","client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:47:22.459409146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.459427614Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.459441823Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.459648322Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"io task","client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.460050723Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"2","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.4600736Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"2","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.460088129Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"0","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.460102037Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"0","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.460720906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"2","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.460756482Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"0","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.461241764Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"2","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.461268082Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"2","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.461284Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.461308768Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"0","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.461323936Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"0","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:22.461337395Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.460709623Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"io task","client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"ds_connection connected from Ok(127.0.0.1:63961)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.460860838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"io task","client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.462276758Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.462305095Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.462891897Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463492217Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463520644Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[0]R flush_numbers[0..12]: [226, 232, 213, 233, 222, 234, 234, 228, 233, 221, 234, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463539342Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[0]R generation[0..12]: [118, 120, 113, 120, 116, 121, 121, 119, 120, 116, 121, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463557241Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463579689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[1]R flush_numbers[0..12]: [226, 232, 213, 233, 222, 234, 234, 228, 233, 221, 234, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463597927Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[1]R generation[0..12]: [118, 120, 113, 120, 116, 121, 121, 119, 120, 116, 121, 114]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463615595Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463633383Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[2]R flush_numbers[0..12]: [226, 232, 213, 233, 222, 234, 234, 228, 233, 221, 234, 216]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463651741Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[2]R generation[0..12]: [118, 120, 113, 120, 116, 121, 121, 119, 120, 116, 121, 114]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:47:32.463669169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463689698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463706186Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463723034Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Next flush: 235","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463739453Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463755671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"":"downstairs","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463772479Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463787698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"0","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463812126Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"1","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463841063Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"client":"2","":"downstairs","session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2024-07-09T21:47:32.463875539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463907616Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"3ccb7a58-9c59-41e5-b18c-b918a008ad12 is now active with session: 61968bbf-8a66-47e7-a4c1-56330ea966d5","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463927714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463943062Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673,"session_id":"61968bbf-8a66-47e7-a4c1-56330ea966d5"} {"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:32.463958531Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 285 len 8192 data: 10 7 02/30 Write block 223 len 4096 data: 11 03/30 Read block 282 len 16384 04/30 Read block 103 len 28672 05/30 Write block 403 len 36864 data: 8 8 7 9 11 13 13 12 10 06/30 Write block 457 len 32768 data: 10 10 8 8 8 12 11 9 07/30 Write block 544 len 40960 data: 6 7 9 9 10 10 12 12 13 12 08/30 Read block 175 len 36864 09/30 Read block 250 len 12288 10/30 Read block 321 len 16384 11/30 Read block 431 len 4096 12/30 Write block 146 len 28672 data: 10 8 6 7 7 8 8 13/30 Flush 14/30 Write block 538 len 40960 data: 9 8 9 9 9 8 7 8 10 10 15/30 Write block 202 len 20480 data: 10 10 11 13 14 16/30 Write block 99 len 4096 data: 11 17/30 Read block 409 len 28672 18/30 Flush 19/30 Flush 20/30 Write block 434 len 16384 data: 14 13 10 10 21/30 Read block 21 len 4096 22/30 Flush 23/30 Write block 434 len 16384 data: 15 14 11 11 24/30 Read block 259 len 28672 25/30 Read block 180 len 32768 26/30 Flush 27/30 Write block 234 len 40960 data: 10 9 10 9 10 6 8 7 7 7 28/30 Write block 259 len 8192 data: 6 4 29/30 Read block 575 len 40960 30/30 Read block 300 len 16384 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 10 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Read 4 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:47:32.492709981Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1673} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 226 226 226 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 120 120 120 233 233 233 F F F 4 080-099 122 122 122 236 236 236 F F F 5 100-119 121 121 121 234 234 234 F F F 6 120-139 121 121 121 234 234 234 F F F 7 140-159 122 122 122 235 235 235 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 116 116 116 221 221 221 F F F 10 200-219 122 122 122 236 236 236 F F F 11 220-239 122 122 122 235 235 235 T T T 12 240-259 114 114 114 216 216 216 T T T 13 260-279 118 118 118 227 227 227 T T T 14 280-299 122 122 122 235 235 235 F F F 15 300-319 121 121 121 234 234 234 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 119 119 119 228 228 228 F F F 18 360-379 121 121 121 234 234 234 F F F 19 380-399 113 113 113 213 213 213 F F F 20 400-419 122 122 122 235 235 235 F F F 21 420-439 122 122 122 239 239 239 F F F 22 440-459 122 122 122 235 235 235 F F F 23 460-479 122 122 122 235 235 235 F F F 24 480-499 120 120 120 232 232 232 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 122 122 122 236 236 236 F F F 27 540-559 122 122 122 236 236 236 F F F 28 560-579 118 118 118 225 225 225 F F F 29 580-599 120 120 120 229 229 229 F F F Max gen: 122, Max flush: 239 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 226 226 226 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 113 113 113 213 213 213 F F F 3 060-079 120 120 120 233 233 233 F F F 4 080-099 122 122 122 236 236 236 F F F 5 100-119 121 121 121 234 234 234 F F F 6 120-139 121 121 121 234 234 234 F F F 7 140-159 122 122 122 235 235 235 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 116 116 116 221 221 221 F F F 10 200-219 122 122 122 236 236 236 F F F 11 220-239 123 123 123 240 240 240 F F F 12 240-259 123 123 123 240 240 240 F F F 13 260-279 123 123 123 240 240 240 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 121 121 121 234 234 234 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 119 119 119 228 228 228 F F F 18 360-379 121 121 121 234 234 234 F F F 19 380-399 113 113 113 213 213 213 F F F 20 400-419 122 122 122 235 235 235 F F F 21 420-439 122 122 122 239 239 239 F F F 22 440-459 122 122 122 235 235 235 F F F 23 460-479 122 122 122 235 235 235 F F F 24 480-499 120 120 120 232 232 232 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 122 122 122 236 236 236 F F F 27 540-559 122 122 122 236 236 236 F F F 28 560-579 118 118 118 225 225 225 F F F 29 580-599 120 120 120 229 229 229 F F F Max gen: 123, Max flush: 240 Begin loop 62 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.691324852Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.691705814Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.691728122Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681} {"msg":"Crucible ca37a47c-f3a1-456b-a5ed-fbd01f289fe7 has session id: 256d12cb-c948-45f6-8bfa-e8745064439a","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692095875Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Upstairs opts: Upstairs UUID: ca37a47c-f3a1-456b-a5ed-fbd01f289fe7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692140841Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Crucible stats registered with UUID: ca37a47c-f3a1-456b-a5ed-fbd01f289fe7","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692161789Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692469518Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681} {"msg":"ca37a47c-f3a1-456b-a5ed-fbd01f289fe7 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692561019Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692578707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"0","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692593486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"0","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692607105Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"1","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692620753Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"1","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692635012Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.69265926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692679278Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"io task","client":"0","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692694386Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"io task","client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692708115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"io task","client":"1","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ds_connection connected from Ok(127.0.0.1:34038)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692863689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"io task","client":"0","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:47:46.692904505Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"io task","client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:47:46.692936492Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"io task","client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:47:46.692966479Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.692990456Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.693016714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ds_connection connected from Ok(127.0.0.1:50134)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.693043561Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"io task","client":"1","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.69315364Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"io task","client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.693683108Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"0","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.693709685Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"0","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.693724694Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"1","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.693739392Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"1","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.694333174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"1","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.694360531Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"0","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.694914586Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"1","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.694942903Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"1","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.694958181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.69497088Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"0","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.694984459Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"0","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:46.694997788Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.693461486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"io task","client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ds_connection connected from Ok(127.0.0.1:46242)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.693588213Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"io task","client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.694588644Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.694617642Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695157708Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695710143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695732951Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[0]R flush_numbers[0..12]: [226, 232, 213, 233, 236, 234, 234, 235, 233, 221, 236, 240]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695751189Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[0]R generation[0..12]: [118, 120, 113, 120, 122, 121, 121, 122, 120, 116, 122, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695768977Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695787295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[1]R flush_numbers[0..12]: [226, 232, 213, 233, 236, 234, 234, 235, 233, 221, 236, 240]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695808664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[1]R generation[0..12]: [118, 120, 113, 120, 122, 121, 121, 122, 120, 116, 122, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695826662Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.69584457Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[2]R flush_numbers[0..12]: [226, 232, 213, 233, 236, 234, 234, 235, 233, 221, 236, 240]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695862098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","Wait for a query_work_queue command to finish before sending IO pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[2]R generation[0..12]: [118, 120, 113, 120, 122, 121, 121, 122, 120, 116, 122, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695882486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695900574Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695916833Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695933511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Next flush: 241","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.69594973Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695966118Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"":"downstairs","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695982896Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.695998015Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"0","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.696015403Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"1","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.696032021Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"client":"2","":"downstairs","session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.69605088Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.696066648Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"ca37a47c-f3a1-456b-a5ed-fbd01f289fe7 is now active with session: 256d12cb-c948-45f6-8bfa-e8745064439a","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.696083096Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.696098195Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681,"session_id":"256d12cb-c948-45f6-8bfa-e8745064439a"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:47:56.696113803Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 111 len 20480 data: 10 11 12 14 13 02/30 Read block 182 len 24576 03/30 Write block 4 len 32768 data: 3 4 5 7 8 9 10 10 04/30 Read block 214 len 32768 05/30 Read block 254 len 8192 06/30 Read block 309 len 28672 07/30 Read block 308 len 8192 08/30 Read block 449 len 20480 09/30 Read block 500 len 8192 10/30 Write block 524 len 24576 data: 9 8 12 11 11 10 11/30 Read block 558 len 8192 12/30 Write block 385 len 32768 data: 13 15 15 16 18 17 18 17 13/30 Read block 439 len 32768 14/30 Write block 251 len 40960 data: 5 5 5 5 8 7 6 6 7 5 15/30 Read block 180 len 40960 16/30 Write block 145 len 20480 data: 9 11 9 7 8 17/30 Flush 18/30 Write block 53 len 12288 data: 11 11 12 19/30 Read block 356 len 36864 20/30 Flush 21/30 Write block 538 len 12288 data: 10 9 10 22/30 Write block 481 len 40960 data: 6 7 8 9 11 12 13 12 11 12 23/30 Read block 146 len 12288 24/30 Read block 227 len 8192 25/30 Flush 26/30 Write block 449 len 8192 data: 14 13 27/30 Write block 571 len 4096 data: 17 28/30 Write block 460 len 16384 data: 9 9 13 12 29/30 Write block 427 len 36864 data: 13 12 12 12 10 11 12 16 15 30/30 Read block 577 len 16384 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Write 1 Done Done Done false 29 Acked 1028 Write 4 Done Done Done false 30 Acked 1029 Write 9 Done Sent Done false 31 Acked 1030 Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 2 1 3 Done 5 3 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:47:56.723277581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1681} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 124 124 124 241 241 241 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 124 124 124 242 242 242 F F F 3 060-079 120 120 120 233 233 233 F F F 4 080-099 122 122 122 236 236 236 F F F 5 100-119 124 124 124 241 241 241 F F F 6 120-139 121 121 121 234 234 234 F F F 7 140-159 124 124 124 241 241 241 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 116 116 116 221 221 221 F F F 10 200-219 122 122 122 236 236 236 F F F 11 220-239 123 123 123 240 240 240 F F F 12 240-259 124 124 124 241 241 241 F F F 13 260-279 124 124 124 241 241 241 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 121 121 121 234 234 234 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 119 119 119 228 228 228 F F F 18 360-379 121 121 121 234 234 234 F F F 19 380-399 124 124 124 241 241 241 F F F 20 400-419 122 122 122 235 235 235 F F F 21 420-439 122 122 122 239 239 239 T T T 22 440-459 122 122 122 235 235 235 T T T 23 460-479 122 122 122 235 235 235 T T T 24 480-499 124 124 124 244 244 244 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 124 124 124 243 243 243 F F F 27 540-559 124 124 124 243 243 243 F F F 28 560-579 118 118 118 225 225 225 T T T 29 580-599 120 120 120 229 229 229 F F F Max gen: 124, Max flush: 244 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 124 124 124 241 241 241 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 124 124 124 242 242 242 F F F 3 060-079 120 120 120 233 233 233 F F F 4 080-099 122 122 122 236 236 236 F F F 5 100-119 124 124 124 241 241 241 F F F 6 120-139 121 121 121 234 234 234 F F F 7 140-159 124 124 124 241 241 241 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 116 116 116 221 221 221 F F F 10 200-219 122 122 122 236 236 236 F F F 11 220-239 123 123 123 240 240 240 F F F 12 240-259 124 124 124 241 241 241 F F F 13 260-279 124 124 124 241 241 241 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 121 121 121 234 234 234 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 119 119 119 228 228 228 F F F 18 360-379 121 121 121 234 234 234 F F F 19 380-399 124 124 124 241 241 241 F F F 20 400-419 122 122 122 235 235 235 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 125 125 125 245 245 245 F F F 23 460-479 125 125 125 245 245 245 F F F 24 480-499 124 124 124 244 244 244 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 124 124 124 243 243 243 F F F 27 540-559 124 124 124 243 243 243 F F F 28 560-579 125 125 125 245 245 245 F F F 29 580-599 120 120 120 229 229 229 F F F Max gen: 125, Max flush: 245 Begin loop 63 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.933010498Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.933479421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.933512558Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689} {"msg":"Crucible a22d8aa7-9a83-45cc-90b0-5db1df778ba4 has session id: 7f748e08-8dd2-42cc-bcff-b9317950fb42","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.933724127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Upstairs opts: Upstairs UUID: a22d8aa7-9a83-45cc-90b0-5db1df778ba4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.933769062Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Crucible stats registered with UUID: a22d8aa7-9a83-45cc-90b0-5db1df778ba4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.93379555Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934062853Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689} {"msg":"a22d8aa7-9a83-45cc-90b0-5db1df778ba4 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934087561Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.93410218Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"0","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934117738Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"0","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934131677Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"1","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934148365Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"1","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934171623Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.93419806Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934223757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"io task","client":"0","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934250455Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"io task","client":"1","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.93430476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"io task","client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"ds_connection connected from Ok(127.0.0.1:59793)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934453915Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"io task","client":"1","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:10.93450233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"io task","client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:10.934533907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"io task","client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"ds_connection connected from Ok(127.0.0.1:49883)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934659654Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"io task","client":"0","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:10.934697621Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934725448Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934744406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.934867974Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"io task","client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.935171603Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"1","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.935197941Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"1","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.935333187Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"0","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.935352936Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"0","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.935770194Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"1","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.935908541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"0","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.93631253Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"1","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.936343318Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"1","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.936361456Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.936373964Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"0","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.936387573Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"0","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:10.936400692Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.936268959Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"io task","client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"ds_connection connected from Ok(127.0.0.1:48452)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.936604196Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"io task","client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.937501507Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.937533664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938004207Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938563282Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938585899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[0]R flush_numbers[0..12]: [241, 232, 242, 233, 236, 241, 234, 241, 233, 221, 236, 240]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938603808Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[0]R generation[0..12]: [124, 120, 124, 120, 122, 124, 121, 124, 120, 116, 122, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938621366Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938639594Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[1]R flush_numbers[0..12]: [241, 232, 242, 233, 236, 241, 234, 241, 233, 221, 236, 240]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938662382Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[1]R generation[0..12]: [124, 120, 124, 120, 122, 124, 121, 124, 120, 116, 122, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.93868004Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938697839Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[2]R flush_numbers[0..12]: [241, 232, 242, 233, 236, 241, 234, 241, 233, 221, 236, 240]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938715027Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[2]R generation[0..12]: [124, 120, 124, 120, 122, 124, 121, 124, 120, 116, 122, 123]","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2024-07-09T21:48:20.938735725Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938754043Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938770541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938793859Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Next flush: 246","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938823146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938847294Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"":"downstairs","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938864482Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.93888517Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"0","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938914837Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"1","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938943014Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"client":"2","":"downstairs","session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938964122Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.938990379Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"a22d8aa7-9a83-45cc-90b0-5db1df778ba4 is now active with session: 7f748e08-8dd2-42cc-bcff-b9317950fb42","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.939019177Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2024-07-09T21:48:20.939045134Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689,"session_id":"7f748e08-8dd2-42cc-bcff-b9317950fb42"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:20.939072581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 192 len 12288 data: 12 13 13 02/30 Read block 567 len 36864 03/30 Write block 50 len 32768 data: 5 8 7 12 12 13 14 14 04/30 Read block 348 len 12288 05/30 Read block 18 len 36864 06/30 Write block 3 len 16384 data: 3 4 5 6 07/30 Write block 580 len 16384 data: 7 8 9 7 08/30 Read block 160 len 40960 09/30 Flush 10/30 Write block 554 len 20480 data: 13 13 15 15 13 11/30 Write block 449 len 24576 data: 15 14 14 12 11 12 12/30 Write block 311 len 8192 data: 10 12 13/30 Read block 312 len 28672 14/30 Write block 344 len 12288 data: 10 9 9 15/30 Write block 543 len 16384 data: 9 8 9 11 16/30 Write block 78 len 4096 data: 9 17/30 Read block 156 len 16384 18/30 Flush 19/30 Write block 210 len 4096 data: 13 20/30 Write block 236 len 4096 data: 11 21/30 Read block 232 len 20480 22/30 Read block 169 len 24576 23/30 Read block 64 len 12288 24/30 Read block 314 len 24576 25/30 Write block 587 len 12288 data: 12 12 13 26/30 Read block 404 len 28672 27/30 Write block 484 len 20480 data: 10 12 13 14 13 28/30 Read block 37 len 16384 29/30 Write block 109 len 36864 data: 11 9 11 12 13 15 14 11 11 30/30 Write block 204 len 36864 data: 12 14 15 16 16 14 14 14 13 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 1 Done Done Done false 20 Acked 1019 Write 1 Done Done Done false 21 Acked 1020 Read 5 Done Done Done false 22 Acked 1021 Read 6 Done Done Done false 23 Acked 1022 Read 3 Done Done Done false 24 Acked 1023 Read 6 Done Done Done false 25 Acked 1024 Write 3 Done Done Done false 26 Acked 1025 Read 7 Done Done Done false 27 Acked 1026 Write 5 Done Done Done false 28 Acked 1027 Read 4 Done Done Done false 29 Acked 1028 Write 9 Done Done Done false 30 Acked 1029 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 11 11 11 33 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:20.964229938Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1689} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 246 246 246 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 126 126 126 246 246 246 F F F 3 060-079 126 126 126 247 247 247 F F F 4 080-099 122 122 122 236 236 236 F F F 5 100-119 124 124 124 241 241 241 T T T 6 120-139 121 121 121 234 234 234 F F F 7 140-159 124 124 124 241 241 241 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 126 126 126 246 246 246 F F F 10 200-219 122 122 122 236 236 236 T T T 11 220-239 123 123 123 240 240 240 T T T 12 240-259 124 124 124 241 241 241 F F F 13 260-279 124 124 124 241 241 241 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 126 126 126 247 247 247 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 126 126 126 247 247 247 F F F 18 360-379 121 121 121 234 234 234 F F F 19 380-399 124 124 124 241 241 241 F F F 20 400-419 122 122 122 235 235 235 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 126 126 126 247 247 247 F F F 23 460-479 125 125 125 245 245 245 F F F 24 480-499 124 124 124 244 244 244 T T T 25 500-519 119 119 119 228 228 228 F F F 26 520-539 124 124 124 243 243 243 F F F 27 540-559 126 126 126 247 247 247 F F F 28 560-579 125 125 125 245 245 245 F F F 29 580-599 126 126 126 246 246 246 T T T Max gen: 126, Max flush: 247 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 246 246 246 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 126 126 126 246 246 246 F F F 3 060-079 126 126 126 247 247 247 F F F 4 080-099 122 122 122 236 236 236 F F F 5 100-119 127 127 127 248 248 248 F F F 6 120-139 121 121 121 234 234 234 F F F 7 140-159 124 124 124 241 241 241 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 126 126 126 246 246 246 F F F 10 200-219 127 127 127 248 248 248 F F F 11 220-239 127 127 127 248 248 248 F F F 12 240-259 124 124 124 241 241 241 F F F 13 260-279 124 124 124 241 241 241 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 126 126 126 247 247 247 F F F 16 320-339 115 115 115 217 217 217 F F F 17 340-359 126 126 126 247 247 247 F F F 18 360-379 121 121 121 234 234 234 F F F 19 380-399 124 124 124 241 241 241 F F F 20 400-419 122 122 122 235 235 235 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 126 126 126 247 247 247 F F F 23 460-479 125 125 125 245 245 245 F F F 24 480-499 127 127 127 248 248 248 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 124 124 124 243 243 243 F F F 27 540-559 126 126 126 247 247 247 F F F 28 560-579 125 125 125 245 245 245 F F F 29 580-599 127 127 127 248 248 248 F F F Max gen: 127, Max flush: 248 Begin loop 64 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.198369505Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.198664846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.198681565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697} {"msg":"Crucible d19698c8-b081-48a7-86d1-101844c0fe02 has session id: 24fd812e-ea33-40b6-96d2-a9d618f3cf5e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199226611Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Upstairs opts: Upstairs UUID: d19698c8-b081-48a7-86d1-101844c0fe02, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199274036Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Crucible stats registered with UUID: d19698c8-b081-48a7-86d1-101844c0fe02","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199298533Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199612432Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697} {"msg":"d19698c8-b081-48a7-86d1-101844c0fe02 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199663997Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199681745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"0","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199696754Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"0","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199710572Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199724821Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.19973827Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"2","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199755618Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"2","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199821421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"io task","client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199845899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"io task","client":"0","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.199861297Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"io task","client":"2","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44154)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.200076146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"io task","client":"2","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:35695)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.200113903Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"io task","client":"0","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:35.200330841Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"io task","client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:35.200363088Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"io task","client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:35.200386296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.200399704Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.200413243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.2005456Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"io task","client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.201019253Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"2","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.201057579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"2","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.201086996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"0","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.201115233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"0","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.201675088Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"2","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.201719184Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"0","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.20226295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"2","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.202290757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"2","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.202306415Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.202319264Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"0","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.202332913Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"0","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:35.202346451Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.202004151Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"io task","client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:33453)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.202433888Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"io task","client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.203429869Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.203454947Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204038889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204611032Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.20463823Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[0]R flush_numbers[0..12]: [246, 232, 246, 247, 236, 248, 234, 241, 233, 246, 248, 248]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204656868Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[0]R generation[0..12]: [126, 120, 126, 126, 122, 127, 121, 124, 120, 126, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204679895Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204698513Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[1]R flush_numbers[0..12]: [246, 232, 246, 247, 236, 248, 234, 241, 233, 246, 248, 248]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204716122Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[1]R generation[0..12]: [126, 120, 126, 126, 122, 127, 121, 124, 120, 126, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.20473373Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204751648Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[2]R flush_numbers[0..12]: [246, 232, 246, 247, 236, 248, 234, 241, 233, 246, 248, 248]","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO time":"2024-07-09T21:48:45.204768926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[2]R generation[0..12]: [126, 120, 126, 126, 122, 127, 121, 124, 120, 126, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204790164Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204807853Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Max found gen is 128","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204823981Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204840709Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Next flush: 249","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204856848Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204873046Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"":"downstairs","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204889494Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204904653Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"0","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204923171Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"1","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.20494008Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"client":"2","":"downstairs","session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204956868Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204972246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"d19698c8-b081-48a7-86d1-101844c0fe02 is now active with session: 24fd812e-ea33-40b6-96d2-a9d618f3cf5e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.204988475Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.205003683Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697,"session_id":"24fd812e-ea33-40b6-96d2-a9d618f3cf5e"} {"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:45.205019112Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 96 len 20480 data: 8 11 10 12 12 02/30 Write block 576 len 4096 data: 15 03/30 Read block 310 len 12288 04/30 Write block 588 len 24576 data: 13 14 14 14 12 14 05/30 Flush 06/30 Write block 1 len 40960 data: 3 3 4 5 6 7 8 9 10 11 07/30 Read block 90 len 16384 08/30 Write block 314 len 40960 data: 13 10 11 14 16 14 13 13 12 11 09/30 Read block 248 len 20480 10/30 Read block 413 len 32768 11/30 Write block 156 len 4096 data: 6 12/30 Write block 395 len 8192 data: 14 11 13/30 Read block 82 len 40960 14/30 Flush 15/30 Write block 487 len 24576 data: 15 14 12 13 12 5 16/30 Read block 263 len 24576 17/30 Write block 357 len 4096 data: 12 18/30 Read block 16 len 28672 19/30 Write block 74 len 40960 data: 11 8 9 9 10 7 9 9 11 11 20/30 Read block 248 len 12288 21/30 Write block 154 len 12288 data: 7 8 7 22/30 Read block 47 len 32768 23/30 Read block 488 len 32768 24/30 Read block 572 len 4096 25/30 Read block 415 len 12288 26/30 Flush 27/30 Write block 452 len 32768 data: 13 12 13 12 10 11 11 9 28/30 Read block 21 len 20480 29/30 Read block 387 len 32768 30/30 Read block 416 len 16384 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 8 Done Done Done false 28 Acked 1027 Read 5 Done Done Done false 29 Acked 1028 Read 8 Done Done Done false 30 Acked 1029 Read 4 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 0 2 Done 3 3 4 10 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:45.233959004Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1697} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 250 250 250 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 126 126 126 246 246 246 F F F 3 060-079 128 128 128 251 251 251 F F F 4 080-099 128 128 128 251 251 251 F F F 5 100-119 128 128 128 249 249 249 F F F 6 120-139 121 121 121 234 234 234 F F F 7 140-159 128 128 128 251 251 251 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 126 126 126 246 246 246 F F F 10 200-219 127 127 127 248 248 248 F F F 11 220-239 127 127 127 248 248 248 F F F 12 240-259 124 124 124 241 241 241 F F F 13 260-279 124 124 124 241 241 241 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 128 128 128 250 250 250 F F F 16 320-339 128 128 128 250 250 250 F F F 17 340-359 128 128 128 251 251 251 F F F 18 360-379 121 121 121 234 234 234 F F F 19 380-399 128 128 128 250 250 250 F F F 20 400-419 122 122 122 235 235 235 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 126 126 126 247 247 247 T T T 23 460-479 125 125 125 245 245 245 F F F 24 480-499 128 128 128 251 251 251 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 124 124 124 243 243 243 F F F 27 540-559 126 126 126 247 247 247 F F F 28 560-579 128 128 128 249 249 249 F F F 29 580-599 128 128 128 249 249 249 F F F Max gen: 128, Max flush: 251 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 250 250 250 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 126 126 126 246 246 246 F F F 3 060-079 128 128 128 251 251 251 F F F 4 080-099 128 128 128 251 251 251 F F F 5 100-119 128 128 128 249 249 249 F F F 6 120-139 121 121 121 234 234 234 F F F 7 140-159 128 128 128 251 251 251 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 126 126 126 246 246 246 F F F 10 200-219 127 127 127 248 248 248 F F F 11 220-239 127 127 127 248 248 248 F F F 12 240-259 124 124 124 241 241 241 F F F 13 260-279 124 124 124 241 241 241 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 128 128 128 250 250 250 F F F 16 320-339 128 128 128 250 250 250 F F F 17 340-359 128 128 128 251 251 251 F F F 18 360-379 121 121 121 234 234 234 F F F 19 380-399 128 128 128 250 250 250 F F F 20 400-419 122 122 122 235 235 235 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 129 129 129 252 252 252 F F F 23 460-479 125 125 125 245 245 245 F F F 24 480-499 128 128 128 251 251 251 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 124 124 124 243 243 243 F F F 27 540-559 126 126 126 247 247 247 F F F 28 560-579 128 128 128 249 249 249 F F F 29 580-599 128 128 128 249 249 249 F F F Max gen: 129, Max flush: 252 Begin loop 65 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.406248113Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.406647293Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.40667757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705} {"msg":"Crucible b3cd1fe7-15f8-444e-9997-8f4514dfa0b8 has session id: a1e48df1-1125-4b1c-b563-fa7a787d7fdd","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407019006Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Upstairs opts: Upstairs UUID: b3cd1fe7-15f8-444e-9997-8f4514dfa0b8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407062272Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Crucible stats registered with UUID: b3cd1fe7-15f8-444e-9997-8f4514dfa0b8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.4070865Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407392869Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705} {"msg":"b3cd1fe7-15f8-444e-9997-8f4514dfa0b8 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407424236Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407438705Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"0","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407454033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"0","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407468092Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"1","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407481821Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"1","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407495029Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407508598Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.40759277Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"io task","client":"1","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407615467Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"io task","client":"0","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407634435Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"io task","client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:55.407841405Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"io task","client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"ds_connection connected from Ok(127.0.0.1:54437)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.407958373Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"io task","client":"1","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"ds_connection connected from Ok(127.0.0.1:47018)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.40799565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"io task","client":"0","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:55.408024347Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"io task","client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:48:55.408054334Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.408077542Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.408102039Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.408125397Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"io task","client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.408622528Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"0","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.408648765Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"0","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.408664053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"1","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.408678232Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"1","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.409309939Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"1","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.409340196Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"0","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.409782733Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"0","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.40981056Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"0","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.409828738Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.409895181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"1","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.409914699Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"1","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:48:55.409929138Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.409785976Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"io task","client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"ds_connection connected from Ok(127.0.0.1:41917)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.410233342Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"io task","client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.411044461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.411071949Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.411672509Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412178559Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412198497Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[0]R flush_numbers[0..12]: [250, 232, 246, 251, 251, 249, 234, 251, 233, 246, 248, 248]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412213095Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[0]R generation[0..12]: [128, 120, 126, 128, 128, 128, 121, 128, 120, 126, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412232164Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412247132Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[1]R flush_numbers[0..12]: [250, 232, 246, 251, 251, 249, 234, 251, 233, 246, 248, 248]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412261381Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[1]R generation[0..12]: [128, 120, 126, 128, 128, 128, 121, 128, 120, 126, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412275529Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412290078Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[2]R flush_numbers[0..12]: [250, 232, 246, 251, 251, 249, 234, 251, 233, 246, 248, 248]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412303977Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[2]R generation[0..12]: [128, 120, 126, 128, 128, 128, 121, 128, 120, 126, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412317925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412333724Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Max found gen is 130","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2024-07-09T21:49:05.412353432Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.4123703Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Next flush: 253","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412386908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412403397Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"":"downstairs","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412420475Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412435664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"0","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412452652Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"1","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.41247161Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"client":"2","":"downstairs","session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412488658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412504197Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"b3cd1fe7-15f8-444e-9997-8f4514dfa0b8 is now active with session: a1e48df1-1125-4b1c-b563-fa7a787d7fdd","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412520355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412535344Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705,"session_id":"a1e48df1-1125-4b1c-b563-fa7a787d7fdd"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:05.412550882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 188 len 24576 data: 8 9 9 13 13 14 02/30 Read block 131 len 32768 03/30 Read block 455 len 12288 04/30 Flush 05/30 Flush 06/30 Write block 187 len 24576 data: 9 9 10 10 14 14 07/30 Write block 134 len 20480 data: 9 8 7 10 10 08/30 Write block 43 len 28672 data: 10 10 9 8 8 8 8 09/30 Read block 412 len 4096 10/30 Read block 247 len 36864 11/30 Read block 188 len 20480 12/30 Read block 255 len 8192 13/30 Write block 547 len 16384 data: 11 11 11 13 14/30 Read block 16 len 12288 15/30 Read block 537 len 32768 16/30 Read block 207 len 16384 17/30 Write block 396 len 20480 data: 12 9 5 7 7 18/30 Write block 440 len 24576 data: 11 13 13 13 13 13 19/30 Read block 115 len 4096 20/30 Write block 342 len 40960 data: 9 10 11 10 10 8 6 6 6 6 21/30 Write block 346 len 32768 data: 11 9 7 7 7 7 6 8 22/30 Write block 546 len 16384 data: 12 12 12 12 23/30 Read block 316 len 36864 24/30 Write block 587 len 28672 data: 13 14 15 15 15 13 15 25/30 Write block 589 len 16384 data: 16 16 16 14 26/30 Write block 371 len 12288 data: 9 10 12 27/30 Read block 542 len 24576 28/30 Write block 405 len 16384 data: 8 10 12 14 29/30 Write block 62 len 20480 data: 14 12 9 11 12 30/30 Read block 2 len 32768 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 8 Done Done Done false 23 Acked 1022 Write 4 Done Done Done false 24 Acked 1023 Read 9 Done Done Done false 25 Acked 1024 Write 7 Done Done Done false 26 Acked 1025 Write 4 Done Done Done false 27 Acked 1026 Write 3 Done Done Done false 28 Acked 1027 Read 6 Done Done Done false 29 Acked 1028 Write 4 Done Done Done false 30 Acked 1029 Write 5 Done Done Done false 31 Acked 1030 Read 8 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 10 10 10 30 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:49:05.438287111Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1705} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 250 250 250 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 130 130 130 255 255 255 F F F 3 060-079 128 128 128 251 251 251 T T T 4 080-099 128 128 128 251 251 251 F F F 5 100-119 128 128 128 249 249 249 F F F 6 120-139 130 130 130 255 255 255 F F F 7 140-159 128 128 128 251 251 251 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 130 130 130 255 255 255 F F F 10 200-219 127 127 127 248 248 248 F F F 11 220-239 127 127 127 248 248 248 F F F 12 240-259 124 124 124 241 241 241 F F F 13 260-279 124 124 124 241 241 241 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 128 128 128 250 250 250 F F F 16 320-339 128 128 128 250 250 250 F F F 17 340-359 130 130 130 255 255 255 T T T 18 360-379 121 121 121 234 234 234 T T T 19 380-399 130 130 130 255 255 255 F F F 20 400-419 130 130 130 255 255 255 T T T 21 420-439 125 125 125 245 245 245 F F F 22 440-459 130 130 130 255 255 255 F F F 23 460-479 125 125 125 245 245 245 F F F 24 480-499 128 128 128 251 251 251 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 124 124 124 243 243 243 F F F 27 540-559 130 130 130 255 255 255 T T T 28 560-579 128 128 128 249 249 249 F F F 29 580-599 128 128 128 249 249 249 T T T Max gen: 130, Max flush: 255 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 250 250 250 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 130 130 130 255 255 255 F F F 3 060-079 131 131 131 256 256 256 F F F 4 080-099 128 128 128 251 251 251 F F F 5 100-119 128 128 128 249 249 249 F F F 6 120-139 130 130 130 255 255 255 F F F 7 140-159 128 128 128 251 251 251 F F F 8 160-179 120 120 120 233 233 233 F F F 9 180-199 130 130 130 255 255 255 F F F 10 200-219 127 127 127 248 248 248 F F F 11 220-239 127 127 127 248 248 248 F F F 12 240-259 124 124 124 241 241 241 F F F 13 260-279 124 124 124 241 241 241 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 128 128 128 250 250 250 F F F 16 320-339 128 128 128 250 250 250 F F F 17 340-359 131 131 131 256 256 256 F F F 18 360-379 131 131 131 256 256 256 F F F 19 380-399 130 130 130 255 255 255 F F F 20 400-419 131 131 131 256 256 256 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 130 130 130 255 255 255 F F F 23 460-479 125 125 125 245 245 245 F F F 24 480-499 128 128 128 251 251 251 F F F 25 500-519 119 119 119 228 228 228 F F F 26 520-539 124 124 124 243 243 243 F F F 27 540-559 131 131 131 256 256 256 F F F 28 560-579 128 128 128 249 249 249 F F F 29 580-599 131 131 131 256 256 256 F F F Max gen: 131, Max flush: 256 Begin loop 66 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.680952316Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.681231288Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.681250556Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713} {"msg":"Crucible 8332f20d-3bd9-410a-9934-5d16c78548d7 has session id: 8eab0214-f72d-4c79-842b-a98540f713a4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.681638188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Upstairs opts: Upstairs UUID: 8332f20d-3bd9-410a-9934-5d16c78548d7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.681673264Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Crucible stats registered with UUID: 8332f20d-3bd9-410a-9934-5d16c78548d7","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.681688923Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682032579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713} {"msg":"8332f20d-3bd9-410a-9934-5d16c78548d7 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682057216Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682071905Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"0","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682087023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"0","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682101092Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"1","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682114641Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"1","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682127869Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682141538Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682236708Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"io task","client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682258846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"io task","client":"0","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682273355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"io task","client":"1","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:33007)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682473375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"io task","client":"1","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:64159)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682611271Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"io task","client":"0","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:49:19.682650747Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"io task","client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:49:19.682670336Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"io task","client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:49:19.682766266Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682783534Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682797053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.682943158Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"io task","client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.683525661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"0","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.683557868Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"0","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.683585185Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"1","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.683612182Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"1","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.684171137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"1","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.684211043Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"0","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.68484109Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"0","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.684865878Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"0","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.684880337Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.684893115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"1","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.684906614Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"1","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:19.684919953Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.684164933Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"io task","client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:46535)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.684510039Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"io task","client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.685594391Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.685622259Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686203651Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686695882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686715131Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[0]R flush_numbers[0..12]: [250, 232, 255, 256, 251, 249, 255, 251, 233, 255, 248, 248]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686734838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[0]R generation[0..12]: [128, 120, 130, 131, 128, 128, 130, 128, 120, 130, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686750407Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686764766Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[1]R flush_numbers[0..12]: [250, 232, 255, 256, 251, 249, 255, 251, 233, 255, 248, 248]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686778724Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[1]R generation[0..12]: [128, 120, 130, 131, 128, 128, 130, 128, 120, 130, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686792663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686806851Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[2]R flush_numbers[0..12]: [250, 232, 255, 256, 251, 249, 255, 251, 233, 255, 248, 248]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.68682075Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[2]R generation[0..12]: [128, 120, 130, 131, 128, 128, 130, 128, 120, 130, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686834549Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686848507Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686861446Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:49:29.686874715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Next flush: 257","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686892223Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686905402Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"":"downstairs","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.68691898Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686933889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"0","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686947528Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"1","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686960786Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"client":"2","":"downstairs","session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686974275Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686986633Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"8332f20d-3bd9-410a-9934-5d16c78548d7 is now active with session: 8eab0214-f72d-4c79-842b-a98540f713a4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.686999622Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.687011671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713,"session_id":"8eab0214-f72d-4c79-842b-a98540f713a4"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:29.68702407Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 487 len 4096 data: 16 02/30 Read block 333 len 8192 03/30 Read block 90 len 28672 04/30 Write block 318 len 24576 data: 17 15 14 14 13 12 05/30 Write block 385 len 32768 data: 14 16 16 17 19 18 19 18 06/30 Read block 587 len 40960 07/30 Write block 394 len 40960 data: 16 15 13 10 6 8 8 6 5 9 08/30 Write block 100 len 32768 data: 13 12 11 14 12 11 12 10 09/30 Read block 258 len 20480 10/30 Read block 576 len 16384 11/30 Write block 203 len 16384 data: 11 13 15 16 12/30 Read block 65 len 32768 13/30 Read block 385 len 40960 14/30 Write block 378 len 16384 data: 5 8 6 8 15/30 Read block 294 len 4096 16/30 Write block 500 len 20480 data: 6 5 5 6 6 17/30 Write block 224 len 12288 data: 11 10 8 18/30 Flush 19/30 Write block 177 len 12288 data: 9 7 6 20/30 Read block 375 len 12288 21/30 Write block 444 len 8192 data: 14 14 22/30 Write block 394 len 4096 data: 17 23/30 Read block 480 len 8192 24/30 Write block 240 len 4096 data: 9 25/30 Write block 162 len 16384 data: 8 9 9 12 26/30 Write block 504 len 40960 data: 7 8 11 10 12 10 12 12 14 11 27/30 Read block 253 len 16384 28/30 Read block 82 len 28672 29/30 Read block 482 len 8192 30/30 Write block 149 len 12288 data: 9 8 9 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 3 Done Done Done false 20 Acked 1019 Read 3 Done Done Done false 21 Acked 1020 Write 2 Done Done Done false 22 Acked 1021 Write 1 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Write 1 Done Done Done false 25 Acked 1024 Write 4 Done Done Done false 26 Acked 1025 Write 10 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Read 7 Done Done Done false 29 Acked 1028 Read 2 Done Done Done false 30 Acked 1029 Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 11 11 11 33 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:49:29.71134001Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1713} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 250 250 250 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 130 130 130 255 255 255 F F F 3 060-079 131 131 131 256 256 256 F F F 4 080-099 128 128 128 251 251 251 F F F 5 100-119 132 132 132 257 257 257 F F F 6 120-139 130 130 130 255 255 255 F F F 7 140-159 128 128 128 251 251 251 T T T 8 160-179 120 120 120 233 233 233 T T T 9 180-199 130 130 130 255 255 255 F F F 10 200-219 132 132 132 257 257 257 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 124 124 124 241 241 241 T T T 13 260-279 124 124 124 241 241 241 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 132 132 132 257 257 257 F F F 16 320-339 132 132 132 257 257 257 F F F 17 340-359 131 131 131 256 256 256 F F F 18 360-379 132 132 132 257 257 257 F F F 19 380-399 132 132 132 257 257 257 T T T 20 400-419 132 132 132 257 257 257 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 130 130 130 255 255 255 T T T 23 460-479 125 125 125 245 245 245 F F F 24 480-499 132 132 132 257 257 257 F F F 25 500-519 132 132 132 257 257 257 T T T 26 520-539 124 124 124 243 243 243 F F F 27 540-559 131 131 131 256 256 256 F F F 28 560-579 128 128 128 249 249 249 F F F 29 580-599 131 131 131 256 256 256 F F F Max gen: 132, Max flush: 257 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 250 250 250 F F F 1 020-039 120 120 120 232 232 232 F F F 2 040-059 130 130 130 255 255 255 F F F 3 060-079 131 131 131 256 256 256 F F F 4 080-099 128 128 128 251 251 251 F F F 5 100-119 132 132 132 257 257 257 F F F 6 120-139 130 130 130 255 255 255 F F F 7 140-159 133 133 133 258 258 258 F F F 8 160-179 133 133 133 258 258 258 F F F 9 180-199 130 130 130 255 255 255 F F F 10 200-219 132 132 132 257 257 257 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 133 133 133 258 258 258 F F F 13 260-279 124 124 124 241 241 241 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 132 132 132 257 257 257 F F F 16 320-339 132 132 132 257 257 257 F F F 17 340-359 131 131 131 256 256 256 F F F 18 360-379 132 132 132 257 257 257 F F F 19 380-399 133 133 133 258 258 258 F F F 20 400-419 132 132 132 257 257 257 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 133 133 133 258 258 258 F F F 23 460-479 125 125 125 245 245 245 F F F 24 480-499 132 132 132 257 257 257 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 124 124 124 243 243 243 F F F 27 540-559 131 131 131 256 256 256 F F F 28 560-579 128 128 128 249 249 249 F F F 29 580-599 131 131 131 256 256 256 F F F Max gen: 133, Max flush: 258 Begin loop 67 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.960050244Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.960622938Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.960651235Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721} {"msg":"Crucible e5f34cd2-0855-4065-ae8f-f63176e76595 has session id: dc2f5843-535b-44b9-a6cc-300c4737c3f3","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.960677302Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Upstairs opts: Upstairs UUID: e5f34cd2-0855-4065-ae8f-f63176e76595, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.960769993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Crucible stats registered with UUID: e5f34cd2-0855-4065-ae8f-f63176e76595","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.96079814Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961112199Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721} {"msg":"e5f34cd2-0855-4065-ae8f-f63176e76595 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961142206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961157185Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961173093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961187152Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"1","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.96120321Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"1","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961231437Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"2","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961258135Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"2","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961284412Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"io task","client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.96130855Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"io task","client":"1","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961332747Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"io task","client":"2","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:49:43.961485322Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"io task","client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:61185)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961570094Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"io task","client":"2","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:61560)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.96160301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"io task","client":"1","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:49:43.961633098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"io task","client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:49:43.96170462Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961735687Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961762315Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.961839967Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"io task","client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.962296102Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"1","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.962321419Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"1","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.962339057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"2","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.962353356Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"2","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.962960996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"1","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.962988203Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"2","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.963498243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"2","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.96352417Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"2","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.963538939Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.963551477Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"1","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.963565266Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"1","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:43.963578395Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.962532734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"io task","client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:48402)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.962755392Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"io task","client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.964099889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.964136115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.964823827Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965358194Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965381192Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[0]R flush_numbers[0..12]: [250, 232, 255, 256, 251, 257, 255, 258, 258, 255, 257, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.96539677Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[0]R generation[0..12]: [128, 120, 130, 131, 128, 132, 130, 133, 133, 130, 132, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965411189Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965425687Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[1]R flush_numbers[0..12]: [250, 232, 255, 256, 251, 257, 255, 258, 258, 255, 257, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965440116Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[1]R generation[0..12]: [128, 120, 130, 131, 128, 132, 130, 133, 133, 130, 132, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965467034Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965482042Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[2]R flush_numbers[0..12]: [250, 232, 255, 256, 251, 257, 255, 258, 258, 255, 257, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.96549618Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[2]R generation[0..12]: [128, 120, 130, 131, 128, 132, 130, 133, 133, 130, 132, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965510149Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965524098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965537247Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965554315Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Next flush: 259","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965567304Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965583512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"":"downstairs","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.96559735Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965612029Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"0","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965636467Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"1","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965659824Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"client":"2","":"downstairs","session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965684372Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.96570513Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"e5f34cd2-0855-4065-ae8f-f63176e76595 is now active with session: dc2f5843-535b-44b9-a6cc-300c4737c3f3","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965726748Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965747816Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721,"session_id":"dc2f5843-535b-44b9-a6cc-300c4737c3f3"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:49:53.965769673Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"pid":1721} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 85 len 20480 data: 11 13 13 13 11 02/30 Flush 03/30 Write block 531 len 28672 data: 8 7 8 7 8 8 8 04/30 Write block 362 len 20480 data: 9 8 7 8 10 05/30 Write block 476 len 28672 data: 6 6 7 7 7 7 8 06/30 Write block 160 len 32768 data: 10 8 9 10 10 13 11 10 07/30 Flush 08/30 Write block 24 len 28672 data: 8 9 9 9 11 13 12 09/30 Write block 119 len 32768 data: 7 6 6 8 12 14 14 14 10/30 Write block 358 len 40960 data: 14 12 11 8 10 9 8 9 11 9 11/30 Write block 120 len 28672 data: 7 7 9 13 15 15 15 12/30 Write block 27 len 40960 data: 10 12 14 13 12 14 13 12 14 15 13/30 Read block 186 len 4096 14/30 Read block 104 len 32768 15/30 Read block 10 len 36864 16/30 Write block 271 len 20480 data: 6 5 7 6 8 17/30 Flush 18/30 Write block 357 len 28672 data: 13 15 13 12 9 11 10 19/30 Flush 20/30 Write block 559 len 24576 data: 11 11 11 11 11 11 21/30 Flush 22/30 Write block 481 len 24576 data: 8 9 9 11 13 14 23/30 Read block 446 len 28672 24/30 Read block 181 len 8192 25/30 Write block 133 len 24576 data: 9 10 9 8 11 11 26/30 Write block 154 len 4096 data: 8 27/30 Read block 477 len 36864 28/30 Read block 239 len 20480 29/30 Read block 144 len 24576 30/30 Write block 11 len 8192 data: 11 12 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 6 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Read 2 Done Done Done false 25 Acked 1024 Write 6 Done Done Done false 26 Acked 1025 Write 1 Done Done Done false 27 Acked 1026 Read 9 Done Done Done false 28 Acked 1027 Read 5 Done Done Done false 29 Acked 1028 Read 6 Done Done Done false 30 Acked 1029 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:49:53.998724887Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1721} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 250 250 250 T T T 1 020-039 134 134 134 261 261 261 F F F 2 040-059 130 130 130 255 255 255 F F F 3 060-079 131 131 131 256 256 256 F F F 4 080-099 134 134 134 259 259 259 F F F 5 100-119 134 134 134 261 261 261 F F F 6 120-139 134 134 134 261 261 261 T T T 7 140-159 133 133 133 258 258 258 T T T 8 160-179 134 134 134 260 260 260 F F F 9 180-199 130 130 130 255 255 255 F F F 10 200-219 132 132 132 257 257 257 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 133 133 133 258 258 258 F F F 13 260-279 134 134 134 261 261 261 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 132 132 132 257 257 257 F F F 16 320-339 132 132 132 257 257 257 F F F 17 340-359 134 134 134 262 262 262 F F F 18 360-379 134 134 134 262 262 262 F F F 19 380-399 133 133 133 258 258 258 F F F 20 400-419 132 132 132 257 257 257 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 133 133 133 258 258 258 F F F 23 460-479 134 134 134 260 260 260 F F F 24 480-499 134 134 134 260 260 260 T T T 25 500-519 133 133 133 258 258 258 F F F 26 520-539 134 134 134 260 260 260 F F F 27 540-559 134 134 134 263 263 263 F F F 28 560-579 134 134 134 263 263 263 F F F 29 580-599 131 131 131 256 256 256 F F F Max gen: 134, Max flush: 263 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 264 264 264 F F F 1 020-039 134 134 134 261 261 261 F F F 2 040-059 130 130 130 255 255 255 F F F 3 060-079 131 131 131 256 256 256 F F F 4 080-099 134 134 134 259 259 259 F F F 5 100-119 134 134 134 261 261 261 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 135 135 135 264 264 264 F F F 8 160-179 134 134 134 260 260 260 F F F 9 180-199 130 130 130 255 255 255 F F F 10 200-219 132 132 132 257 257 257 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 133 133 133 258 258 258 F F F 13 260-279 134 134 134 261 261 261 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 132 132 132 257 257 257 F F F 16 320-339 132 132 132 257 257 257 F F F 17 340-359 134 134 134 262 262 262 F F F 18 360-379 134 134 134 262 262 262 F F F 19 380-399 133 133 133 258 258 258 F F F 20 400-419 132 132 132 257 257 257 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 133 133 133 258 258 258 F F F 23 460-479 134 134 134 260 260 260 F F F 24 480-499 135 135 135 264 264 264 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 134 134 134 260 260 260 F F F 27 540-559 134 134 134 263 263 263 F F F 28 560-579 134 134 134 263 263 263 F F F 29 580-599 131 131 131 256 256 256 F F F Max gen: 135, Max flush: 264 Begin loop 68 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.212966398Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.213259379Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.213286666Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729} {"msg":"Crucible f1ae7a05-0fc1-4d9f-93bd-7ec1de00bde4 has session id: 0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.213597085Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Upstairs opts: Upstairs UUID: f1ae7a05-0fc1-4d9f-93bd-7ec1de00bde4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.213640411Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Crucible stats registered with UUID: f1ae7a05-0fc1-4d9f-93bd-7ec1de00bde4","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.213665509Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.213966459Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729} {"msg":"f1ae7a05-0fc1-4d9f-93bd-7ec1de00bde4 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.21405055Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214083057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214110564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214136092Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"1","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214163459Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"1","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214187607Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"2","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214213324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"2","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214237182Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"io task","client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214261879Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"io task","client":"1","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214285327Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"io task","client":"2","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"ds_connection connected from Ok(127.0.0.1:62939)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214455Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"io task","client":"1","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"ds_connection connected from Ok(127.0.0.1:50917)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214492416Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"io task","client":"2","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:08.214597396Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"io task","client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:08.214643421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"io task","client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:08.214674589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214695387Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214720464Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.214746411Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"io task","client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.215211535Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"2","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.215237243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"2","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.215251921Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"1","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.21526592Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"1","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.215850432Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"1","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.215878069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"2","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.216392968Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"1","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.216419545Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"1","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.216434304Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.216446663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"2","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.216460101Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"2","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:08.2164735Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.215863696Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"io task","client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"ds_connection connected from Ok(127.0.0.1:57830)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.216119801Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"io task","client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.217158358Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.217184255Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.217721782Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218259218Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218286106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[0]R flush_numbers[0..12]: [264, 261, 255, 256, 259, 261, 264, 264, 260, 255, 257, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218304874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[0]R generation[0..12]: [135, 134, 130, 131, 134, 134, 135, 135, 134, 130, 132, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218322842Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.21834063Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[1]R flush_numbers[0..12]: [264, 261, 255, 256, 259, 261, 264, 264, 260, 255, 257, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218358209Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[1]R generation[0..12]: [135, 134, 130, 131, 134, 134, 135, 135, 134, 130, 132, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218375547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218393465Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[2]R flush_numbers[0..12]: [264, 261, 255, 256, 259, 261, 264, 264, 260, 255, 257, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218412093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[2]R generation[0..12]: [135, 134, 130, 131, 134, 134, 135, 135, 134, 130, 132, 132]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:50:18.21844176Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218462718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218479327Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218500954Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Next flush: 265","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218518993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218535391Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"":"downstairs","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218552439Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218567578Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"0","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218584386Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"1","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218600935Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"client":"2","":"downstairs","session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218617983Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218633581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"f1ae7a05-0fc1-4d9f-93bd-7ec1de00bde4 is now active with session: 0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.21864984Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218664828Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729,"session_id":"0b13789d-9370-4cb5-8e65-2c5a5c2ebb2f"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:18.218680197Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 531 len 16384 data: 9 8 9 8 02/30 Read block 162 len 16384 03/30 Flush 04/30 Flush 05/30 Flush 06/30 Write block 384 len 20480 data: 9 15 17 17 18 07/30 Flush 08/30 Write block 406 len 16384 data: 11 13 15 14 09/30 Write block 34 len 16384 data: 13 15 16 18 10/30 Read block 311 len 24576 11/30 Flush 12/30 Write block 410 len 12288 data: 13 11 10 13/30 Read block 257 len 32768 14/30 Write block 194 len 28672 data: 14 11 8 10 11 11 11 15/30 Write block 44 len 20480 data: 11 10 9 9 9 16/30 Write block 111 len 4096 data: 12 17/30 Read block 135 len 28672 18/30 Flush 19/30 Write block 144 len 32768 data: 10 10 12 10 8 10 9 10 20/30 Read block 314 len 20480 21/30 Read block 441 len 36864 22/30 Flush 23/30 Write block 376 len 20480 data: 12 7 6 9 7 24/30 Read block 416 len 20480 25/30 Read block 585 len 36864 26/30 Write block 61 len 4096 data: 13 27/30 Write block 175 len 4096 data: 9 28/30 Read block 274 len 32768 29/30 Read block 446 len 20480 30/30 Write block 596 len 12288 data: 11 9 7 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 5 Done Done Done false 25 Acked 1024 Read 5 Done Done Done false 26 Acked 1025 Read 9 Done Done Done false 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Write 1 Done Done Done false 29 Acked 1028 Read 8 Done Done Done false 30 Acked 1029 Read 5 Done Done Done false 31 Acked 1030 Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:18.248878834Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1729} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 264 264 264 F F F 1 020-039 136 136 136 269 269 269 F F F 2 040-059 136 136 136 271 271 271 F F F 3 060-079 131 131 131 256 256 256 T T T 4 080-099 134 134 134 259 259 259 F F F 5 100-119 136 136 136 271 271 271 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 136 136 136 272 272 272 F F F 8 160-179 134 134 134 260 260 260 T T T 9 180-199 136 136 136 271 271 271 F F F 10 200-219 136 136 136 271 271 271 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 133 133 133 258 258 258 F F F 13 260-279 134 134 134 261 261 261 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 132 132 132 257 257 257 F F F 16 320-339 132 132 132 257 257 257 F F F 17 340-359 134 134 134 262 262 262 F F F 18 360-379 134 134 134 262 262 262 T T T 19 380-399 136 136 136 268 268 268 T T T 20 400-419 136 136 136 270 270 270 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 133 133 133 258 258 258 F F F 23 460-479 134 134 134 260 260 260 F F F 24 480-499 135 135 135 264 264 264 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 136 136 136 265 265 265 F F F 27 540-559 134 134 134 263 263 263 F F F 28 560-579 134 134 134 263 263 263 F F F 29 580-599 131 131 131 256 256 256 T T T Max gen: 136, Max flush: 272 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 264 264 264 F F F 1 020-039 136 136 136 269 269 269 F F F 2 040-059 136 136 136 271 271 271 F F F 3 060-079 137 137 137 273 273 273 F F F 4 080-099 134 134 134 259 259 259 F F F 5 100-119 136 136 136 271 271 271 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 136 136 136 272 272 272 F F F 8 160-179 137 137 137 273 273 273 F F F 9 180-199 136 136 136 271 271 271 F F F 10 200-219 136 136 136 271 271 271 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 133 133 133 258 258 258 F F F 13 260-279 134 134 134 261 261 261 F F F 14 280-299 122 122 122 235 235 235 F F F 15 300-319 132 132 132 257 257 257 F F F 16 320-339 132 132 132 257 257 257 F F F 17 340-359 134 134 134 262 262 262 F F F 18 360-379 137 137 137 273 273 273 F F F 19 380-399 137 137 137 273 273 273 F F F 20 400-419 136 136 136 270 270 270 F F F 21 420-439 125 125 125 245 245 245 F F F 22 440-459 133 133 133 258 258 258 F F F 23 460-479 134 134 134 260 260 260 F F F 24 480-499 135 135 135 264 264 264 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 136 136 136 265 265 265 F F F 27 540-559 134 134 134 263 263 263 F F F 28 560-579 134 134 134 263 263 263 F F F 29 580-599 137 137 137 273 273 273 F F F Max gen: 137, Max flush: 273 Begin loop 69 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.480004081Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.480270584Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.480288313Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737} {"msg":"Crucible 03a27519-ff0a-4d50-a6af-9f1aa0a86402 has session id: 84cb1e3f-cd2a-4d53-98ed-40bab84f27da","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.480635668Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Upstairs opts: Upstairs UUID: 03a27519-ff0a-4d50-a6af-9f1aa0a86402, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.480675664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Crucible stats registered with UUID: 03a27519-ff0a-4d50-a6af-9f1aa0a86402","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.480691113Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.480990523Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737} {"msg":"03a27519-ff0a-4d50-a6af-9f1aa0a86402 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481074355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481099652Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"0","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481116671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"0","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481130859Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481148417Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481172715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"2","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481194933Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"2","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481208982Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"io task","client":"0","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.48122317Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"io task","client":"2","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481237069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"io task","client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"ds_connection connected from Ok(127.0.0.1:44329)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481396063Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"io task","client":"2","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"ds_connection connected from Ok(127.0.0.1:38493)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481574055Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"io task","client":"0","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:32.481616411Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"io task","client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:32.481648598Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"io task","client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:32.481678305Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481699573Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.48172378Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.481781495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"io task","client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.482278076Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"0","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.482311142Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"0","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.482370476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"2","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.482402363Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"2","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.482893595Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"2","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.482921082Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"0","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.483413183Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"0","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.483451099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"0","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.483478926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.483507234Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"2","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.483535301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"2","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:32.483556499Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.482346314Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"io task","client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"ds_connection connected from Ok(127.0.0.1:51125)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.48248155Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"io task","client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.48369469Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.483736266Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484248445Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484767144Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484791881Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[0]R flush_numbers[0..12]: [264, 269, 271, 273, 259, 271, 264, 272, 273, 271, 271, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484810449Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[0]R generation[0..12]: [135, 136, 136, 137, 134, 136, 135, 136, 137, 136, 136, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484828708Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484846566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[1]R flush_numbers[0..12]: [264, 269, 271, 273, 259, 271, 264, 272, 273, 271, 271, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484863994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[1]R generation[0..12]: [135, 136, 136, 137, 134, 136, 135, 136, 137, 136, 136, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484881202Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484898661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[2]R flush_numbers[0..12]: [264, 269, 271, 273, 259, 271, 264, 272, 273, 271, 271, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484916129Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[2]R generation[0..12]: [135, 136, 136, 137, 134, 136, 135, 136, 137, 136, 136, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484934787Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484956255Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484972833Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.484993162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Next flush: 274","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.48500993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.485026578Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"":"downstairs","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.485043137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.485058225Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"0","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.485074793Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"1","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.485091212Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"client":"2","":"downstairs","session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.48510796Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.485124918Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"03a27519-ff0a-4d50-a6af-9f1aa0a86402 is now active with session: 84cb1e3f-cd2a-4d53-98ed-40bab84f27da","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.485141257Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.485156065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737,"session_id":"84cb1e3f-cd2a-4d53-98ed-40bab84f27da"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:42.485171474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 422 len 8192 data: 10 9 02/30 Write block 250 len 16384 data: 4 6 6 6 03/30 Read block 533 len 16384 04/30 Flush 05/30 Write block 463 len 4096 data: 13 06/30 Flush 07/30 Write block 451 len 24576 data: 15 14 13 14 13 11 08/30 Write block 248 len 8192 data: 6 4 09/30 Write block 19 len 16384 data: 7 8 11 8 10/30 Read block 165 len 24576 11/30 Read block 510 len 4096 12/30 Write block 211 len 32768 data: 15 14 13 12 12 11 14 10 13/30 Write block 195 len 24576 data: 12 9 11 12 12 12 14/30 Read block 539 len 12288 15/30 Write block 372 len 40960 data: 11 13 14 14 13 8 7 10 8 9 16/30 Write block 304 len 40960 data: 10 12 12 10 10 11 11 11 13 12 17/30 Write block 140 len 24576 data: 11 10 11 11 11 11 18/30 Flush 19/30 Write block 381 len 4096 data: 10 20/30 Write block 390 len 28672 data: 19 20 19 16 18 16 14 21/30 Write block 547 len 24576 data: 13 13 13 14 13 14 22/30 Read block 332 len 24576 23/30 Flush 24/30 Write block 364 len 40960 data: 9 10 12 10 10 15 12 10 12 14 25/30 Read block 91 len 8192 26/30 Read block 491 len 16384 27/30 Write block 66 len 36864 data: 13 10 11 12 13 14 12 11 12 28/30 Read block 307 len 8192 29/30 Write block 337 len 40960 data: 8 7 8 8 9 10 11 12 11 12 30/30 Write block 290 len 20480 data: 9 9 8 7 8 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 10 Done Done Done false 25 Acked 1024 Read 2 Done Done Done false 26 Acked 1025 Read 4 Done Done Done false 27 Acked 1026 Write 9 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Write 10 Sent Done Sent false 30 Acked 1029 Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 1 2 5 Done 5 6 5 16 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:42.514794038Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1737} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 276 276 276 F F F 1 020-039 138 138 138 276 276 276 F F F 2 040-059 136 136 136 271 271 271 F F F 3 060-079 137 137 137 273 273 273 T T T 4 080-099 134 134 134 259 259 259 F F F 5 100-119 136 136 136 271 271 271 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 137 137 137 273 273 273 F F F 9 180-199 138 138 138 276 276 276 F F F 10 200-219 138 138 138 276 276 276 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 138 138 138 276 276 276 F F F 13 260-279 134 134 134 261 261 261 F F F 14 280-299 122 122 122 235 235 235 T T T 15 300-319 138 138 138 276 276 276 F F F 16 320-339 132 132 132 257 257 257 T T T 17 340-359 134 134 134 262 262 262 T T T 18 360-379 138 138 138 276 276 276 T T T 19 380-399 138 138 138 277 277 277 F F F 20 400-419 136 136 136 270 270 270 F F F 21 420-439 138 138 138 274 274 274 F F F 22 440-459 138 138 138 276 276 276 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 135 135 135 264 264 264 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 136 136 136 265 265 265 F F F 27 540-559 138 138 138 277 277 277 F F F 28 560-579 134 134 134 263 263 263 F F F 29 580-599 137 137 137 273 273 273 F F F Max gen: 138, Max flush: 277 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 276 276 276 F F F 1 020-039 138 138 138 276 276 276 F F F 2 040-059 136 136 136 271 271 271 F F F 3 060-079 139 139 139 278 278 278 F F F 4 080-099 134 134 134 259 259 259 F F F 5 100-119 136 136 136 271 271 271 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 137 137 137 273 273 273 F F F 9 180-199 138 138 138 276 276 276 F F F 10 200-219 138 138 138 276 276 276 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 138 138 138 276 276 276 F F F 13 260-279 134 134 134 261 261 261 F F F 14 280-299 139 139 139 278 278 278 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 139 139 139 278 278 278 F F F 18 360-379 139 139 139 278 278 278 F F F 19 380-399 138 138 138 277 277 277 F F F 20 400-419 136 136 136 270 270 270 F F F 21 420-439 138 138 138 274 274 274 F F F 22 440-459 138 138 138 276 276 276 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 135 135 135 264 264 264 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 136 136 136 265 265 265 F F F 27 540-559 138 138 138 277 277 277 F F F 28 560-579 134 134 134 263 263 263 F F F 29 580-599 137 137 137 273 273 273 F F F Max gen: 139, Max flush: 278 Begin loop 70 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.741949589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.742193204Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.742209433Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745} {"msg":"Crucible 1cbfb931-57ab-4891-8527-bdb9acc11683 has session id: 3baeabc2-1180-4cc1-a48a-cca793d6690e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.742631981Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Upstairs opts: Upstairs UUID: 1cbfb931-57ab-4891-8527-bdb9acc11683, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.742666297Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Crucible stats registered with UUID: 1cbfb931-57ab-4891-8527-bdb9acc11683","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.742683946Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743028102Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745} {"msg":"1cbfb931-57ab-4891-8527-bdb9acc11683 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743058908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743073397Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"0","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743088156Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"0","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743105664Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"1","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743128022Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"1","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743154519Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743179916Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743205424Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"io task","client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743231261Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"io task","client":"0","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743256769Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"io task","client":"1","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"ds_connection connected from Ok(127.0.0.1:52263)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743412324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"io task","client":"0","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"ds_connection connected from Ok(127.0.0.1:33138)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743592176Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"io task","client":"1","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:56.743636681Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"io task","client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:56.743665898Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"io task","client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:50:56.743685716Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743707084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743730002Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.743883367Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"io task","client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.744214984Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"1","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.744240471Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"1","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.74435689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"0","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.744375948Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"0","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.74475658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"1","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.744885937Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"0","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.74526395Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"1","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.745291237Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"1","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.745307006Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.745319524Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"0","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.745332863Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"0","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:50:56.745346032Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.744921408Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"io task","client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"ds_connection connected from Ok(127.0.0.1:45273)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.745106379Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"io task","client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.746327249Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.746359026Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.746900872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747440728Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747472375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[0]R flush_numbers[0..12]: [276, 276, 271, 278, 259, 271, 264, 276, 273, 276, 276, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747492593Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[0]R generation[0..12]: [138, 138, 136, 139, 134, 136, 135, 138, 137, 138, 138, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747511001Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747529919Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[1]R flush_numbers[0..12]: [276, 276, 271, 278, 259, 271, 264, 276, 273, 276, 276, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747547548Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[1]R generation[0..12]: [138, 138, 136, 139, 134, 136, 135, 138, 137, 138, 138, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747565096Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747582974Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[2]R flush_numbers[0..12]: [276, 276, 271, 278, 259, 271, 264, 276, 273, 276, 276, 257]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:51:06.747600502Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[2]R generation[0..12]: [138, 138, 136, 139, 134, 136, 135, 138, 137, 138, 138, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747618361Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747635969Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Max found gen is 140","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747655967Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747673035Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Next flush: 279","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747689403Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747705672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"":"downstairs","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.74772267Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747737969Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"0","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747754717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"1","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747771366Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"client":"2","":"downstairs","session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747788024Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747803472Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"1cbfb931-57ab-4891-8527-bdb9acc11683 is now active with session: 3baeabc2-1180-4cc1-a48a-cca793d6690e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747819571Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747834529Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745,"session_id":"3baeabc2-1180-4cc1-a48a-cca793d6690e"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:06.747850168Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 358 len 8192 data: 16 14 02/30 Write block 200 len 36864 data: 13 9 11 12 14 16 17 17 17 03/30 Write block 567 len 40960 data: 7 12 11 13 18 15 15 15 13 16 04/30 Write block 186 len 24576 data: 7 10 10 11 11 15 05/30 Read block 175 len 4096 06/30 Write block 527 len 24576 data: 12 12 11 8 10 9 07/30 Write block 580 len 40960 data: 8 9 10 8 10 9 10 14 15 17 08/30 Read block 230 len 12288 09/30 Read block 385 len 16384 10/30 Write block 8 len 28672 data: 10 11 12 12 13 13 14 11/30 Write block 8 len 40960 data: 11 12 13 13 14 14 15 14 14 13 12/30 Read block 217 len 20480 13/30 Write block 570 len 16384 data: 14 19 16 16 14/30 Read block 44 len 36864 15/30 Read block 334 len 40960 16/30 Read block 278 len 12288 17/30 Read block 586 len 36864 18/30 Read block 63 len 28672 19/30 Write block 188 len 12288 data: 11 12 12 20/30 Read block 585 len 20480 21/30 Flush 22/30 Write block 260 len 8192 data: 6 4 23/30 Read block 372 len 32768 24/30 Write block 205 len 40960 data: 17 18 18 18 15 15 16 15 14 13 25/30 Write block 268 len 20480 data: 5 7 5 7 6 26/30 Flush 27/30 Write block 112 len 12288 data: 13 14 16 28/30 Read block 204 len 20480 29/30 Write block 175 len 24576 data: 10 7 10 8 7 7 30/30 Write block 169 len 8192 data: 6 4 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 3 Done Done Done false 28 Acked 1027 Read 5 Done Done Done false 29 Acked 1028 Write 6 Sent Sent Sent false 30 Acked 1029 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 2 2 6 Done 2 2 2 6 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:51:06.774816935Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1745} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 279 279 279 F F F 1 020-039 138 138 138 276 276 276 F F F 2 040-059 136 136 136 271 271 271 F F F 3 060-079 139 139 139 278 278 278 F F F 4 080-099 134 134 134 259 259 259 F F F 5 100-119 136 136 136 271 271 271 T T T 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 137 137 137 273 273 273 T T T 9 180-199 140 140 140 279 279 279 T T T 10 200-219 140 140 140 280 280 280 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 138 138 138 276 276 276 F F F 13 260-279 140 140 140 280 280 280 F F F 14 280-299 139 139 139 278 278 278 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 140 140 140 279 279 279 F F F 18 360-379 139 139 139 278 278 278 F F F 19 380-399 138 138 138 277 277 277 F F F 20 400-419 136 136 136 270 270 270 F F F 21 420-439 138 138 138 274 274 274 F F F 22 440-459 138 138 138 276 276 276 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 135 135 135 264 264 264 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 140 140 140 279 279 279 F F F 27 540-559 138 138 138 277 277 277 F F F 28 560-579 140 140 140 279 279 279 F F F 29 580-599 140 140 140 279 279 279 F F F Max gen: 140, Max flush: 280 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 279 279 279 F F F 1 020-039 138 138 138 276 276 276 F F F 2 040-059 136 136 136 271 271 271 F F F 3 060-079 139 139 139 278 278 278 F F F 4 080-099 134 134 134 259 259 259 F F F 5 100-119 141 141 141 281 281 281 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 141 141 141 281 281 281 F F F 10 200-219 140 140 140 280 280 280 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 138 138 138 276 276 276 F F F 13 260-279 140 140 140 280 280 280 F F F 14 280-299 139 139 139 278 278 278 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 140 140 140 279 279 279 F F F 18 360-379 139 139 139 278 278 278 F F F 19 380-399 138 138 138 277 277 277 F F F 20 400-419 136 136 136 270 270 270 F F F 21 420-439 138 138 138 274 274 274 F F F 22 440-459 138 138 138 276 276 276 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 135 135 135 264 264 264 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 140 140 140 279 279 279 F F F 27 540-559 138 138 138 277 277 277 F F F 28 560-579 140 140 140 279 279 279 F F F 29 580-599 140 140 140 279 279 279 F F F Max gen: 141, Max flush: 281 Begin loop 71 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.982381898Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.982754661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.982775829Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753} {"msg":"Crucible 1c5591d5-8cfc-4be2-9ead-8a4ca1b62efd has session id: 8748bb79-6d97-4a0e-92f0-c6a76780dc39","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983211376Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Upstairs opts: Upstairs UUID: 1c5591d5-8cfc-4be2-9ead-8a4ca1b62efd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983260181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Crucible stats registered with UUID: 1c5591d5-8cfc-4be2-9ead-8a4ca1b62efd","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983287798Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.98357464Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753} {"msg":"1c5591d5-8cfc-4be2-9ead-8a4ca1b62efd active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983644833Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983664341Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"0","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983683409Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"0","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983697638Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983711146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983724865Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"2","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983748033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"2","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983771891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"io task","client":"0","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983796648Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"io task","client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.983814076Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"io task","client":"2","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:51:20.98397679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"io task","client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:51:20.984015786Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"io task","client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:51:20.984043153Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.984063962Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.984085889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"ds_connection connected from Ok(127.0.0.1:45688)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.98417364Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"io task","client":"0","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"ds_connection connected from Ok(127.0.0.1:43345)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.984213767Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"io task","client":"2","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.984243354Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"io task","client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.984868532Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"2","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.984895909Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"2","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.984910967Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"0","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.984931536Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"0","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.985512198Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"2","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.985539375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"0","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.986044795Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"0","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.986067403Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"0","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.986081351Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.98609349Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"2","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.986106949Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"2","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:20.986120128Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.985588106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"io task","client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"ds_connection connected from Ok(127.0.0.1:47152)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.98585328Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"io task","client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.986887418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.986913755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.98746756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988050162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.9880758Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[0]R flush_numbers[0..12]: [279, 276, 271, 278, 259, 281, 264, 276, 281, 281, 280, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988092148Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[0]R generation[0..12]: [140, 138, 136, 139, 134, 141, 135, 138, 141, 141, 140, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988106946Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988121755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[1]R flush_numbers[0..12]: [279, 276, 271, 278, 259, 281, 264, 276, 281, 281, 280, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988135864Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[1]R generation[0..12]: [140, 138, 136, 139, 134, 141, 135, 138, 141, 141, 140, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988149752Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988163891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[2]R flush_numbers[0..12]: [279, 276, 271, 278, 259, 281, 264, 276, 281, 281, 280, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.98817797Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[2]R generation[0..12]: [140, 138, 136, 139, 134, 141, 135, 138, 141, 141, 140, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988191998Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988206137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 142","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988227655Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988242283Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Next flush: 282","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988255442Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.98826843Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"":"downstairs","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988281719Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988293838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"0","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988307327Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"1","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988320425Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"client":"2","":"downstairs","session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988333944Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988347053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"1c5591d5-8cfc-4be2-9ead-8a4ca1b62efd is now active with session: 8748bb79-6d97-4a0e-92f0-c6a76780dc39","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988359941Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.98837185Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753,"session_id":"8748bb79-6d97-4a0e-92f0-c6a76780dc39"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:30.988384369Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 420 len 12288 data: 11 10 11 02/30 Read block 135 len 40960 03/30 Read block 373 len 36864 04/30 Write block 256 len 20480 data: 8 7 7 8 7 05/30 Read block 394 len 4096 06/30 Flush 07/30 Write block 34 len 4096 data: 14 08/30 Write block 279 len 8192 data: 8 9 09/30 Flush 10/30 Write block 403 len 28672 data: 10 9 9 12 14 16 15 11/30 Read block 24 len 12288 12/30 Flush 13/30 Write block 554 len 20480 data: 14 14 16 16 14 14/30 Read block 595 len 20480 15/30 Flush 16/30 Write block 356 len 28672 data: 11 14 17 15 13 10 12 17/30 Read block 249 len 32768 18/30 Read block 514 len 36864 19/30 Write block 350 len 28672 data: 8 8 7 9 6 7 12 20/30 Read block 390 len 12288 21/30 Read block 185 len 20480 22/30 Read block 554 len 32768 23/30 Write block 417 len 40960 data: 9 9 9 12 11 12 10 9 10 11 24/30 Read block 51 len 24576 25/30 Read block 9 len 12288 26/30 Read block 296 len 8192 27/30 Read block 152 len 16384 28/30 Read block 156 len 40960 29/30 Write block 217 len 12288 data: 15 11 11 30/30 Write block 568 len 28672 data: 13 12 15 20 17 17 16 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:15 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 16 Acked 1015 Write 7 Done Done Done false 17 Acked 1016 Read 8 Done Done Done false 18 Acked 1017 Read 9 Done Done Done false 19 Acked 1018 Write 7 Done Done Done false 20 Acked 1019 Read 3 Done Done Done false 21 Acked 1020 Read 5 Done Done Done false 22 Acked 1021 Read 8 Done Done Done false 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Read 6 Done Done Done false 25 Acked 1024 Read 3 Done Done Done false 26 Acked 1025 Read 2 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Write 3 Done Done Done false 30 Acked 1029 Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 14 14 14 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1014 1014 1014 Downstairs last five completed: 1014 1013 1012 1011 1010 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:51:31.017788455Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1753} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 279 279 279 F F F 1 020-039 142 142 142 283 283 283 F F F 2 040-059 136 136 136 271 271 271 F F F 3 060-079 139 139 139 278 278 278 F F F 4 080-099 134 134 134 259 259 259 F F F 5 100-119 141 141 141 281 281 281 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 141 141 141 281 281 281 F F F 10 200-219 140 140 140 280 280 280 T T T 11 220-239 132 132 132 257 257 257 F F F 12 240-259 142 142 142 282 282 282 F F F 13 260-279 142 142 142 283 283 283 F F F 14 280-299 142 142 142 283 283 283 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 140 140 140 279 279 279 T T T 18 360-379 139 139 139 278 278 278 T T T 19 380-399 138 138 138 277 277 277 F F F 20 400-419 142 142 142 284 284 284 T T T 21 420-439 142 142 142 282 282 282 T T T 22 440-459 138 138 138 276 276 276 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 135 135 135 264 264 264 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 140 140 140 279 279 279 F F F 27 540-559 142 142 142 285 285 285 F F F 28 560-579 140 140 140 279 279 279 T T T 29 580-599 140 140 140 279 279 279 F F F Max gen: 142, Max flush: 285 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 279 279 279 F F F 1 020-039 142 142 142 283 283 283 F F F 2 040-059 136 136 136 271 271 271 F F F 3 060-079 139 139 139 278 278 278 F F F 4 080-099 134 134 134 259 259 259 F F F 5 100-119 141 141 141 281 281 281 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 141 141 141 281 281 281 F F F 10 200-219 143 143 143 286 286 286 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 142 142 142 282 282 282 F F F 13 260-279 142 142 142 283 283 283 F F F 14 280-299 142 142 142 283 283 283 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 143 143 143 286 286 286 F F F 18 360-379 143 143 143 286 286 286 F F F 19 380-399 138 138 138 277 277 277 F F F 20 400-419 143 143 143 286 286 286 F F F 21 420-439 143 143 143 286 286 286 F F F 22 440-459 138 138 138 276 276 276 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 135 135 135 264 264 264 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 140 140 140 279 279 279 F F F 27 540-559 142 142 142 285 285 285 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 140 140 140 279 279 279 F F F Max gen: 143, Max flush: 286 Begin loop 72 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.256971024Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.257237797Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.257254756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761} {"msg":"Crucible ea8f3a08-b64f-4e75-9c13-2554401f6eab has session id: 71b1607d-863d-4229-82af-34ee5be1a3b0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.257668244Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Upstairs opts: Upstairs UUID: ea8f3a08-b64f-4e75-9c13-2554401f6eab, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.25771521Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Crucible stats registered with UUID: ea8f3a08-b64f-4e75-9c13-2554401f6eab","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.257736878Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.25801717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761} {"msg":"ea8f3a08-b64f-4e75-9c13-2554401f6eab active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258058286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258074944Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"0","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258090683Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"0","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258104661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"1","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.25811842Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"1","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258131769Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258145207Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258211761Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"io task","client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258230299Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"io task","client":"0","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258244067Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"io task","client":"1","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:51:45.25842462Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"io task","client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:51:45.258468955Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"io task","client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ds_connection connected from Ok(127.0.0.1:57617)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258570645Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"io task","client":"1","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:51:45.258599852Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258615621Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258632629Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.258659266Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"io task","client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ds_connection connected from Ok(127.0.0.1:39767)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.2587195Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"io task","client":"0","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.259180815Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"0","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.259206202Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"0","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.259221051Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"1","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.259234539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"1","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.259801883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"1","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.259942749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"0","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.260379736Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"1","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.260411683Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"1","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.260429911Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.260445379Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"0","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.260462178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"0","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:45.260478716Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.260270963Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"io task","client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ds_connection connected from Ok(127.0.0.1:42873)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.260378992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"io task","client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.261362325Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.261396551Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.261955856Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.26251512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262535648Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[0]R flush_numbers[0..12]: [279, 283, 271, 278, 259, 281, 264, 276, 281, 281, 286, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262562455Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[0]R generation[0..12]: [140, 142, 136, 139, 134, 141, 135, 138, 141, 141, 143, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262576764Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262592012Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[1]R flush_numbers[0..12]: [279, 283, 271, 278, 259, 281, 264, 276, 281, 281, 286, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262606201Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[1]R generation[0..12]: [140, 142, 136, 139, 134, 141, 135, 138, 141, 141, 143, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.26262021Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262635008Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[2]R flush_numbers[0..12]: [279, 283, 271, 278, 259, 281, 264, 276, 281, 281, 286, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262649077Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[2]R generation[0..12]: [140, 142, 136, 139, 134, 141, 135, 138, 141, 141, 143, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262662845Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262680164Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Max found gen is 144","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262693642Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Generation requested: 144 >= found:144","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262707041Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msgWait for a query_work_queue command to finish before sending IO ":"Next flush: 287","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.26272351Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262736718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"":"downstairs","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262749927Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262761906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"0","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262775234Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"1","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262788363Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"client":"2","":"downstairs","session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262801502Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262813711Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"ea8f3a08-b64f-4e75-9c13-2554401f6eab is now active with session: 71b1607d-863d-4229-82af-34ee5be1a3b0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262826659Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262838558Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761,"session_id":"71b1607d-863d-4229-82af-34ee5be1a3b0"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:51:55.262850787Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 553 len 12288 data: 13 15 15 02/30 Write block 81 len 4096 data: 10 03/30 Read block 123 len 40960 04/30 Write block 535 len 28672 data: 9 9 9 11 10 11 10 05/30 Read block 374 len 20480 06/30 Read block 494 len 20480 07/30 Write block 276 len 20480 data: 7 7 8 9 10 08/30 Write block 440 len 16384 data: 12 14 14 14 09/30 Write block 103 len 36864 data: 15 13 12 13 11 12 12 10 13 10/30 Write block 84 len 24576 data: 12 12 14 14 14 12 11/30 Read block 86 len 20480 12/30 Write block 538 len 28672 data: 12 11 12 11 10 10 9 13/30 Read block 302 len 32768 14/30 Write block 448 len 40960 data: 14 16 15 16 15 14 15 14 12 12 15/30 Read block 32 len 40960 16/30 Read block 272 len 4096 17/30 Flush 18/30 Write block 552 len 28672 data: 15 14 16 16 17 17 15 19/30 Write block 118 len 8192 data: 7 8 20/30 Read block 297 len 16384 21/30 Read block 80 len 24576 22/30 Write block 384 len 20480 data: 10 16 18 18 19 23/30 Write block 544 len 40960 data: 10 10 13 14 14 14 15 14 16 15 24/30 Write block 241 len 20480 data: 8 8 8 6 5 25/30 Flush 26/30 Write block 23 len 40960 data: 8 9 10 10 11 13 15 14 13 15 27/30 Read block 197 len 32768 28/30 Read block 342 len 8192 29/30 Read block 207 len 4096 30/30 Write block 480 len 4096 data: 8 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Write 10 Done Done Done false 27 Acked 1026 Read 8 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Read 1 Done Done Done false 30 Acked 1029 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:51:55.29026204Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1761} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 279 279 279 F F F 1 020-039 142 142 142 283 283 283 T T T 2 040-059 136 136 136 271 271 271 F F F 3 060-079 139 139 139 278 278 278 F F F 4 080-099 144 144 144 287 287 287 F F F 5 100-119 144 144 144 288 288 288 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 141 141 141 281 281 281 F F F 10 200-219 143 143 143 286 286 286 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 144 144 144 288 288 288 F F F 13 260-279 144 144 144 287 287 287 F F F 14 280-299 144 144 144 287 287 287 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 143 143 143 286 286 286 F F F 18 360-379 143 143 143 286 286 286 F F F 19 380-399 144 144 144 288 288 288 F F F 20 400-419 143 143 143 286 286 286 F F F 21 420-439 143 143 143 286 286 286 F F F 22 440-459 144 144 144 287 287 287 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 135 135 135 264 264 264 T T T 25 500-519 133 133 133 258 258 258 F F F 26 520-539 144 144 144 287 287 287 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 140 140 140 279 279 279 F F F Max gen: 144, Max flush: 288 A Difference in extent metadata is expected here On loop 72 Verifying data now No change after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 279 279 279 F F F 1 020-039 145 145 145 289 289 289 F F F 2 040-059 136 136 136 271 271 271 F F F 3 060-079 139 139 139 278 278 278 F F F 4 080-099 144 144 144 287 287 287 F F F 5 100-119 144 144 144 288 288 288 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 141 141 141 281 281 281 F F F 10 200-219 143 143 143 286 286 286 F F F 11 220-239 132 132 132 257 257 257 F F F 12 240-259 144 144 144 288 288 288 F F F 13 260-279 144 144 144 287 287 287 F F F 14 280-299 144 144 144 287 287 287 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 143 143 143 286 286 286 F F F 18 360-379 143 143 143 286 286 286 F F F 19 380-399 144 144 144 288 288 288 F F F 20 400-419 143 143 143 286 286 286 F F F 21 420-439 143 143 143 286 286 286 F F F 22 440-459 144 144 144 287 287 287 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 145 145 145 289 289 289 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 144 144 144 287 287 287 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 140 140 140 279 279 279 F F F Max gen: 145, Max flush: 289 Begin loop 73 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.478258732Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.478638724Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.478665491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769} {"msg":"Crucible 1dcfc160-5f45-4171-9b93-52ca42c87e25 has session id: bb9c89da-b5af-47e3-b779-f03965373349","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.47908498Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Upstairs opts: Upstairs UUID: 1dcfc160-5f45-4171-9b93-52ca42c87e25, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479133335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Crucible stats registered with UUID: 1dcfc160-5f45-4171-9b93-52ca42c87e25","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479161582Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479429466Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769} {"msg":"1dcfc160-5f45-4171-9b93-52ca42c87e25 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479544314Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479574691Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479601019Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479625336Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"1","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479649264Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"1","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479669532Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"2","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.47969332Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"2","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479719107Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"io task","client":"2","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479744644Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"io task","client":"1","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479768382Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"io task","client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"ds_connection connected from Ok(127.0.0.1:51123)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.47988764Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"io task","client":"2","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"ds_connection connected from Ok(127.0.0.1:39623)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.479923267Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"io task","client":"1","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:09.479952314Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"io task","client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:09.479978191Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"io task","client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:09.480006338Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.480033406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.480058663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.480277411Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"io task","client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.480707179Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"1","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.480732626Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"1","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.480747855Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"2","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.480762523Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"2","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.481296021Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"2","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.481408139Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"1","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.481739027Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"2","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.481761064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"2","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.481775233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.48190759Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"1","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.481925478Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"1","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:09.481938947Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.480988846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"io task","client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"ds_connection connected from Ok(127.0.0.1:55635)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.481180127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"io task","client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.482519484Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.482558Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483033533Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.48356838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483593537Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[0]R flush_numbers[0..12]: [279, 289, 271, 278, 287, 288, 264, 276, 281, 281, 286, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483612875Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[0]R generation[0..12]: [140, 145, 136, 139, 144, 144, 135, 138, 141, 141, 143, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483630624Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483648602Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[1]R flush_numbers[0..12]: [279, 289, 271, 278, 287, 288, 264, 276, 281, 281, 286, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.48366625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[1]R generation[0..12]: [140, 145, 136, 139, 144, 144, 135, 138, 141, 141, 143, 132]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483683479Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483701017Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[2]R flush_numbers[0..12]: [279, 289, 271, 278, 287, 288, 264, 276, 281, 281, 286, 257]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483718445Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[2]R generation[0..12]: [140, 145, 136, 139, 144, 144, 135, 138, 141, 141, 143, 132]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:52:19.483740023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483758291Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Max found gen is 146","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483774679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Generation requested: 146 >= found:146","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483791438Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Next flush: 290","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483807646Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483823895Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"":"downstairs","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483840603Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483855731Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"0","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.48387251Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"1","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483890208Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"client":"2","":"downstairs","session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483907117Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483922425Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"1dcfc160-5f45-4171-9b93-52ca42c87e25 is now active with session: bb9c89da-b5af-47e3-b779-f03965373349","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483938693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.483953602Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769,"session_id":"bb9c89da-b5af-47e3-b779-f03965373349"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:19.48396899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 41 len 28672 data: 8 8 11 12 11 10 10 02/30 Read block 288 len 28672 03/30 Write block 385 len 12288 data: 17 19 19 04/30 Read block 280 len 28672 05/30 Flush 06/30 Write block 183 len 12288 data: 8 6 7 07/30 Flush 08/30 Write block 290 len 20480 data: 10 10 9 8 9 09/30 Read block 580 len 16384 10/30 Flush 11/30 Write block 380 len 24576 data: 9 11 6 8 11 18 12/30 Write block 240 len 28672 data: 10 9 9 9 7 6 5 13/30 Flush 14/30 Write block 40 len 24576 data: 10 9 9 12 13 12 15/30 Read block 7 len 4096 16/30 Flush 17/30 Write block 228 len 16384 data: 4 6 7 8 18/30 Read block 184 len 28672 19/30 Read block 7 len 8192 20/30 Write block 482 len 32768 data: 10 10 12 14 15 17 15 13 21/30 Write block 360 len 24576 data: 14 11 13 11 10 11 22/30 Read block 377 len 24576 23/30 Read block 566 len 40960 24/30 Read block 585 len 28672 25/30 Read block 492 len 28672 26/30 Write block 256 len 32768 data: 9 8 8 9 8 5 8 11 27/30 Read block 80 len 28672 28/30 Read block 150 len 4096 29/30 Read block 293 len 36864 30/30 Read block 41 len 20480 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 17 Acked 1016 Write 4 Done Done Done false 18 Acked 1017 Read 7 Done Done Done false 19 Acked 1018 Read 2 Done Done Done false 20 Acked 1019 Write 8 Done Done Done false 21 Acked 1020 Write 6 Done Done Done false 22 Acked 1021 Read 6 Done Done Done false 23 Acked 1022 Read 10 Done Done Done false 24 Acked 1023 Read 7 Done Done Done false 25 Acked 1024 Read 7 Done Done Done false 26 Acked 1025 Write 8 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Read 5 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 14 14 14 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1015 1015 1015 Downstairs last five completed: 1015 1014 1013 1012 1011 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:19.514488475Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1769} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 279 279 279 F F F 1 020-039 145 145 145 289 289 289 F F F 2 040-059 146 146 146 294 294 294 F F F 3 060-079 139 139 139 278 278 278 F F F 4 080-099 144 144 144 287 287 287 F F F 5 100-119 144 144 144 288 288 288 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 146 146 146 291 291 291 F F F 10 200-219 143 143 143 286 286 286 F F F 11 220-239 132 132 132 257 257 257 T T T 12 240-259 146 146 146 293 293 293 T T T 13 260-279 144 144 144 287 287 287 T T T 14 280-299 146 146 146 292 292 292 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 143 143 143 286 286 286 F F F 18 360-379 143 143 143 286 286 286 T T T 19 380-399 146 146 146 293 293 293 F F F 20 400-419 143 143 143 286 286 286 F F F 21 420-439 143 143 143 286 286 286 F F F 22 440-459 144 144 144 287 287 287 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 145 145 145 289 289 289 T T T 25 500-519 133 133 133 258 258 258 F F F 26 520-539 144 144 144 287 287 287 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 140 140 140 279 279 279 F F F Max gen: 146, Max flush: 294 A Difference in extent metadata is expected here On loop 73 Verifying data now No change after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 279 279 279 F F F 1 020-039 145 145 145 289 289 289 F F F 2 040-059 146 146 146 294 294 294 F F F 3 060-079 139 139 139 278 278 278 F F F 4 080-099 144 144 144 287 287 287 F F F 5 100-119 144 144 144 288 288 288 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 146 146 146 291 291 291 F F F 10 200-219 143 143 143 286 286 286 F F F 11 220-239 147 147 147 295 295 295 F F F 12 240-259 147 147 147 295 295 295 F F F 13 260-279 147 147 147 295 295 295 F F F 14 280-299 146 146 146 292 292 292 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 143 143 143 286 286 286 F F F 18 360-379 147 147 147 295 295 295 F F F 19 380-399 146 146 146 293 293 293 F F F 20 400-419 143 143 143 286 286 286 F F F 21 420-439 143 143 143 286 286 286 F F F 22 440-459 144 144 144 287 287 287 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 147 147 147 295 295 295 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 144 144 144 287 287 287 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 140 140 140 279 279 279 F F F Max gen: 147, Max flush: 295 Begin loop 74 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.740665693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.740965443Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.74099303Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777} {"msg":"Crucible fb2f7e25-a9de-4d7f-a295-0fc5329ea6d0 has session id: a5b8dbf8-81dc-45ba-9d83-1da4ca786e10","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741320698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Upstairs opts: Upstairs UUID: fb2f7e25-a9de-4d7f-a295-0fc5329ea6d0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741359354Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Crucible stats registered with UUID: fb2f7e25-a9de-4d7f-a295-0fc5329ea6d0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741379142Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741712059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777} {"msg":"fb2f7e25-a9de-4d7f-a295-0fc5329ea6d0 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741820018Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741845236Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"0","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741861334Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"0","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741875423Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"1","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741889331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"1","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.74190301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741916689Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741930177Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"io task","client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741943926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"io task","client":"1","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.741957565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"io task","client":"0","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"ds_connection connected from Ok(127.0.0.1:46826)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.742123438Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"io task","client":"1","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"ds_connection connected from Ok(127.0.0.1:64498)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.742162004Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"io task","client":"0","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:33.742190712Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"io task","client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:33.742222688Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"io task","client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:33.742251766Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.742316649Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.742345866Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.742371874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"io task","client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.742931008Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"0","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.742957936Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"0","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.742973444Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"1","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.742987682Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"1","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.743574744Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"1","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.743610561Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"0","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.744141428Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"1","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.744166946Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"1","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.744181194Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.744193673Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"0","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.744210081Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"0","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:33.74422379Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.744109116Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"io task","client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"ds_connection connected from Ok(127.0.0.1:61450)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.744511157Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"io task","client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.74548369Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.745511427Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746038135Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746561623Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746584051Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[0]R flush_numbers[0..12]: [279, 289, 294, 278, 287, 288, 264, 276, 281, 291, 286, 295]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746602239Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[0]R generation[0..12]: [140, 145, 146, 139, 144, 144, 135, 138, 141, 146, 143, 147]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746620077Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746637926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[1]R flush_numbers[0..12]: [279, 289, 294, 278, 287, 288, 264, 276, 281, 291, 286, 295]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746655734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[1]R generation[0..12]: [140, 145, 146, 139, 144, 144, 135, 138, 141, 146, 143, 147]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746673482Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746691301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[2]R flush_numbers[0..12]: [279, 289, 294, 278, 287, 288, 264, 276, 281, 291, 286, 295]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746712548Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[2]R generation[0..12]: [140, 145, 146, 139, 144, 144, 135, 138, 141, 146, 143, 147]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746730286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2024-07-09T21:52:43.746747815Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Max found gen is 148","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746767773Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Generation requested: 148 >= found:148","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746784391Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Next flush: 296","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.74680093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746817258Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"":"downstairs","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746834056Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746849015Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"0","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746865793Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"1","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746882282Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"client":"2","":"downstairs","session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.74689982Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746913088Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"fb2f7e25-a9de-4d7f-a295-0fc5329ea6d0 is now active with session: a5b8dbf8-81dc-45ba-9d83-1da4ca786e10","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746926137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746938206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777,"session_id":"a5b8dbf8-81dc-45ba-9d83-1da4ca786e10"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:43.746952264Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 75 len 16384 data: 9 10 10 11 03/30 Read block 579 len 32768 04/30 Read block 459 len 24576 05/30 Read block 515 len 12288 06/30 Read block 352 len 40960 07/30 Flush 08/30 Write block 449 len 4096 data: 17 09/30 Write block 219 len 32768 data: 12 8 11 13 12 12 11 9 10/30 Read block 27 len 20480 11/30 Read block 344 len 24576 12/30 Write block 62 len 28672 data: 15 13 10 12 14 11 12 13/30 Read block 521 len 40960 14/30 Read block 560 len 32768 15/30 Write block 404 len 24576 data: 10 10 13 15 17 16 16/30 Flush 17/30 Write block 591 len 24576 data: 17 15 16 15 13 12 18/30 Write block 92 len 12288 data: 10 10 10 19/30 Flush 20/30 Flush 21/30 Write block 204 len 16384 data: 15 18 19 19 22/30 Flush 23/30 Write block 217 len 16384 data: 16 12 13 9 24/30 Flush 25/30 Write block 398 len 32768 data: 7 9 9 7 6 11 11 11 26/30 Write block 371 len 40960 data: 11 13 15 15 15 14 9 8 11 10 27/30 Write block 77 len 40960 data: 11 12 8 10 11 12 12 13 13 15 28/30 Write block 279 len 12288 data: 10 11 10 29/30 Read block 516 len 40960 30/30 Read block 243 len 8192 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 8 Done Done Done false 26 Acked 1025 Write 10 Done Done Done false 27 Acked 1026 Write 10 Done Done Done false 28 Acked 1027 Write 3 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Read 2 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 6 6 6 18 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:43.778857862Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1777} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 279 279 279 F F F 1 020-039 145 145 145 289 289 289 F F F 2 040-059 146 146 146 294 294 294 F F F 3 060-079 148 148 148 298 298 298 T T T 4 080-099 148 148 148 299 299 299 T T T 5 100-119 144 144 144 288 288 288 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 146 146 146 291 291 291 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 148 148 148 302 302 302 F F F 12 240-259 147 147 147 295 295 295 F F F 13 260-279 147 147 147 295 295 295 T T T 14 280-299 146 146 146 292 292 292 T T T 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 143 143 143 286 286 286 F F F 18 360-379 147 147 147 295 295 295 T T T 19 380-399 146 146 146 293 293 293 T T T 20 400-419 148 148 148 298 298 298 T T T 21 420-439 143 143 143 286 286 286 F F F 22 440-459 148 148 148 298 298 298 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 147 147 147 295 295 295 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 144 144 144 287 287 287 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 148 148 148 299 299 299 F F F Max gen: 148, Max flush: 302 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 279 279 279 F F F 1 020-039 145 145 145 289 289 289 F F F 2 040-059 146 146 146 294 294 294 F F F 3 060-079 149 149 149 303 303 303 F F F 4 080-099 149 149 149 303 303 303 F F F 5 100-119 144 144 144 288 288 288 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 146 146 146 291 291 291 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 148 148 148 302 302 302 F F F 12 240-259 147 147 147 295 295 295 F F F 13 260-279 149 149 149 303 303 303 F F F 14 280-299 149 149 149 303 303 303 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 143 143 143 286 286 286 F F F 18 360-379 149 149 149 303 303 303 F F F 19 380-399 149 149 149 303 303 303 F F F 20 400-419 149 149 149 303 303 303 F F F 21 420-439 143 143 143 286 286 286 F F F 22 440-459 148 148 148 298 298 298 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 147 147 147 295 295 295 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 144 144 144 287 287 287 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 148 148 148 299 299 299 F F F Max gen: 149, Max flush: 303 Begin loop 75 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.024042163Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.024414576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.024433564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785} {"msg":"Crucible f1852b2a-1e9c-4910-8f7c-ec58c5459ceb has session id: fb430341-cce2-43a7-91ac-78f70fd7b7e9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.02478054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Upstairs opts: Upstairs UUID: f1852b2a-1e9c-4910-8f7c-ec58c5459ceb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.024824955Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Crucible stats registered with UUID: f1852b2a-1e9c-4910-8f7c-ec58c5459ceb","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.024851913Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025123566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785} {"msg":"f1852b2a-1e9c-4910-8f7c-ec58c5459ceb active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025206028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025231695Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"0","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025250763Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"0","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025275661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025301838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025328016Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"2","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025354143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"2","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025377461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"io task","client":"2","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025401548Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"io task","client":"0","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025424666Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"io task","client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:58.025543324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"io task","client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:55705)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025632635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"io task","client":"2","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:58.025771412Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"io task","client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:52:58.025808028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:36537)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025832926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"io task","client":"0","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025855113Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.025869612Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.02588605Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"io task","client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.026313148Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"0","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.026340055Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"0","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.026361483Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"2","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.02638754Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"2","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.026989671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"2","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.027041286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"0","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.027534487Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"2","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.027560574Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"2","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.027575143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.027588072Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"0","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.02760507Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"0","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:52:58.027619198Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.027278838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"io task","client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:62100)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.02766072Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"io task","client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.028602127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.028628754Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029135184Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029650833Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029673121Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[0]R flush_numbers[0..12]: [279, 289, 294, 303, 303, 288, 264, 276, 281, 291, 302, 302]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029688009Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[0]R generation[0..12]: [140, 145, 146, 149, 149, 144, 135, 138, 141, 146, 148, 148]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029702438Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029716757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[1]R flush_numbers[0..12]: [279, 289, 294, 303, 303, 288, 264, 276, 281, 291, 302, 302]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029731465Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[1]R generation[0..12]: [140, 145, 146, 149, 149, 144, 135, 138, 141, 146, 148, 148]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029746764Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029761182Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[2]R flush_numbers[0..12]: [279, 289, 294, 303, 303, 288, 264, 276, 281, 291, 302, 302]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.02977959Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[2]R generation[0..12]: [140, 145, 146, 149, 149, 144, 135, 138, 141, 146, 148, 148]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029802078Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:53:08.029827126Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Max found gen is 150","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029850053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Generation requested: 150 >= found:150","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029872771Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Next flush: 304","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029893399Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029916257Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"":"downstairs","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029941064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029957853Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"0","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.029978491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"1","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.030002668Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"client":"2","":"downstairs","session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.030027656Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.030050193Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"f1852b2a-1e9c-4910-8f7c-ec58c5459ceb is now active with session: fb430341-cce2-43a7-91ac-78f70fd7b7e9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.030064562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.030076811Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785,"session_id":"fb430341-cce2-43a7-91ac-78f70fd7b7e9"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:08.03009178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 486 len 16384 data: 16 18 16 14 02/30 Read block 231 len 24576 03/30 Read block 284 len 4096 04/30 Write block 17 len 24576 data: 14 10 8 9 12 9 05/30 Read block 493 len 24576 06/30 Read block 79 len 24576 07/30 Read block 300 len 32768 08/30 Read block 122 len 16384 09/30 Read block 32 len 36864 10/30 Read block 457 len 24576 11/30 Read block 106 len 12288 12/30 Read block 547 len 8192 13/30 Write block 52 len 12288 data: 8 13 13 14/30 Read block 357 len 32768 15/30 Read block 468 len 24576 16/30 Flush 17/30 Write block 8 len 28672 data: 12 13 14 14 15 15 16 18/30 Write block 283 len 32768 data: 9 10 11 8 7 10 12 11 19/30 Read block 192 len 8192 20/30 Read block 392 len 4096 21/30 Read block 283 len 16384 22/30 Read block 133 len 28672 23/30 Write block 64 len 20480 data: 11 13 15 12 13 24/30 Read block 169 len 20480 25/30 Read block 101 len 28672 26/30 Read block 258 len 12288 27/30 Read block 498 len 20480 28/30 Read block 317 len 12288 29/30 Read block 596 len 16384 30/30 Write block 286 len 4096 data: 9 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 17 Acked 1016 Write 7 Done Done Done false 18 Acked 1017 Write 8 Done Done Done false 19 Acked 1018 Read 2 Done Done Done false 20 Acked 1019 Read 1 Done Done Done false 21 Acked 1020 Read 4 Done Done Done false 22 Acked 1021 Read 7 Done Done Done false 23 Acked 1022 Write 5 Done Done Done false 24 Acked 1023 Read 5 Done Done Done false 25 Acked 1024 Read 7 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Read 4 Done Sent Done false 30 Acked 1029 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 2 1 4 Done 13 12 13 38 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1015 1015 1015 Downstairs last five completed: 1015 1014 1013 1012 1011 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:53:08.054124918Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1785} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 150 150 150 304 304 304 T T T 1 020-039 150 150 150 304 304 304 F F F 2 040-059 150 150 150 304 304 304 F F F 3 060-079 149 149 149 303 303 303 T T T 4 080-099 149 149 149 303 303 303 F F F 5 100-119 144 144 144 288 288 288 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 146 146 146 291 291 291 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 148 148 148 302 302 302 F F F 12 240-259 147 147 147 295 295 295 F F F 13 260-279 149 149 149 303 303 303 F F F 14 280-299 149 149 149 303 303 303 T T T 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 143 143 143 286 286 286 F F F 18 360-379 149 149 149 303 303 303 F F F 19 380-399 149 149 149 303 303 303 F F F 20 400-419 149 149 149 303 303 303 F F F 21 420-439 143 143 143 286 286 286 F F F 22 440-459 148 148 148 298 298 298 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 150 150 150 304 304 304 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 144 144 144 287 287 287 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 148 148 148 299 299 299 F F F Max gen: 150, Max flush: 304 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 150 150 150 304 304 304 F F F 3 060-079 151 151 151 305 305 305 F F F 4 080-099 149 149 149 303 303 303 F F F 5 100-119 144 144 144 288 288 288 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 141 141 141 281 281 281 F F F 9 180-199 146 146 146 291 291 291 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 148 148 148 302 302 302 F F F 12 240-259 147 147 147 295 295 295 F F F 13 260-279 149 149 149 303 303 303 F F F 14 280-299 151 151 151 305 305 305 F F F 15 300-319 138 138 138 276 276 276 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 143 143 143 286 286 286 F F F 18 360-379 149 149 149 303 303 303 F F F 19 380-399 149 149 149 303 303 303 F F F 20 400-419 149 149 149 303 303 303 F F F 21 420-439 143 143 143 286 286 286 F F F 22 440-459 148 148 148 298 298 298 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 150 150 150 304 304 304 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 144 144 144 287 287 287 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 148 148 148 299 299 299 F F F Max gen: 151, Max flush: 305 Begin loop 76 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.255799562Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.25622014Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.256239348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793} {"msg":"Crucible 787aef98-7080-4a23-bef9-715934b132f7 has session id: c9d3f75c-2e5a-42d2-8130-9c917371376c","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.256546058Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Upstairs opts: Upstairs UUID: 787aef98-7080-4a23-bef9-715934b132f7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.256583084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Crucible stats registered with UUID: 787aef98-7080-4a23-bef9-715934b132f7","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.256599932Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.256940059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793} {"msg":"787aef98-7080-4a23-bef9-715934b132f7 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.256962096Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.256975505Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.256989784Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.25702915Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"1","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257057017Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"1","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257081824Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"2","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257105452Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"2","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257204072Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"io task","client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257237089Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"io task","client":"1","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257261307Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"io task","client":"2","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"ds_connection connected from Ok(127.0.0.1:45345)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257381515Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"io task","client":"1","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"ds_connection connected from Ok(127.0.0.1:59254)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257417111Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"io task","client":"2","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:53:22.25752891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"io task","client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:53:22.257562307Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"io task","client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:53:22.257601753Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257622531Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257644219Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.257762077Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"io task","client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.258106243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"2","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.25813282Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"2","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.258148499Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"1","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.258162708Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"1","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.258782986Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"2","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.258809703Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"1","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.259311124Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"2","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.259337501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"2","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.259353119Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.259369108Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"1","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.259382956Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"1","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:22.259396115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.259650206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"io task","client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"ds_connection connected from Ok(127.0.0.1:38446)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.259915679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"io task","client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.261110361Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.261142708Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.261691054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.26222871Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262252708Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[0]R flush_numbers[0..12]: [305, 304, 304, 305, 303, 288, 264, 276, 281, 291, 302, 302]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262270846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[0]R generation[0..12]: [151, 150, 150, 151, 149, 144, 135, 138, 141, 146, 148, 148]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262288894Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262306973Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[1]R flush_numbers[0..12]: [305, 304, 304, 305, 303, 288, 264, 276, 281, 291, 302, 302]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262324551Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[1]R generation[0..12]: [151, 150, 150, 151, 149, 144, 135, 138, 141, 146, 148, 148]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262346569Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262364627Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[2]R flush_numbers[0..12]: [305, 304, 304, 305, 303, 288, 264, 276, 281, 291, 302, 302]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262382155Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} Wait for a query_work_queue command to finish before sending IO {"msg":"[2]R generation[0..12]: [151, 150, 150, 151, 149, 144, 135, 138, 141, 146, 148, 148]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262402813Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262421161Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.26243727Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262454068Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Next flush: 306","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262470376Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262486565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"":"downstairs","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262503123Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262518202Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"0","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.26253473Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"1","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262551248Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"client":"2","":"downstairs","session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262568187Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262580506Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"787aef98-7080-4a23-bef9-715934b132f7 is now active with session: c9d3f75c-2e5a-42d2-8130-9c917371376c","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262595504Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262607703Z","Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793,"session_id":"c9d3f75c-2e5a-42d2-8130-9c917371376c"} {"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:32.262624011Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 348 len 32768 data: 8 8 9 9 8 10 7 8 02/30 Write block 593 len 8192 data: 17 16 03/30 Read block 219 len 16384 04/30 Read block 50 len 8192 05/30 Write block 176 len 16384 data: 8 11 9 8 06/30 Read block 539 len 12288 07/30 Read block 153 len 4096 08/30 Read block 155 len 16384 09/30 Write block 241 len 16384 data: 10 10 10 8 10/30 Write block 316 len 8192 data: 12 15 11/30 Write block 435 len 20480 data: 16 12 12 12 9 12/30 Read block 386 len 8192 13/30 Read block 90 len 24576 14/30 Read block 301 len 16384 15/30 Write block 304 len 28672 data: 11 13 13 11 11 12 12 16/30 Read block 502 len 16384 17/30 Read block 488 len 36864 18/30 Read block 374 len 16384 19/30 Read block 269 len 16384 20/30 Read block 242 len 4096 21/30 Read block 121 len 4096 22/30 Read block 415 len 32768 23/30 Read block 392 len 24576 24/30 Write block 397 len 16384 data: 11 8 10 10 25/30 Read block 595 len 8192 26/30 Flush 27/30 Write block 487 len 36864 data: 19 17 15 14 13 6 6 6 5 28/30 Read block 0 len 16384 29/30 Write block 386 len 20480 data: 20 20 20 20 20 30/30 Read block 571 len 36864 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Read 4 Done Done Done false 30 Acked 1029 Write 5 Done Done Done false 31 Acked 1030 Read 9 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:53:32.288207415Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1793} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 150 150 150 304 304 304 F F F 3 060-079 151 151 151 305 305 305 F F F 4 080-099 149 149 149 303 303 303 F F F 5 100-119 144 144 144 288 288 288 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 152 152 152 306 306 306 F F F 9 180-199 146 146 146 291 291 291 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 148 148 148 302 302 302 F F F 12 240-259 152 152 152 306 306 306 F F F 13 260-279 149 149 149 303 303 303 F F F 14 280-299 151 151 151 305 305 305 F F F 15 300-319 152 152 152 306 306 306 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 152 152 152 306 306 306 F F F 18 360-379 149 149 149 303 303 303 F F F 19 380-399 152 152 152 307 307 307 T T T 20 400-419 152 152 152 307 307 307 F F F 21 420-439 152 152 152 306 306 306 F F F 22 440-459 148 148 148 298 298 298 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 150 150 150 304 304 304 T T T 25 500-519 133 133 133 258 258 258 F F F 26 520-539 144 144 144 287 287 287 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 152 152 152 306 306 306 F F F Max gen: 152, Max flush: 307 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 150 150 150 304 304 304 F F F 3 060-079 151 151 151 305 305 305 F F F 4 080-099 149 149 149 303 303 303 F F F 5 100-119 144 144 144 288 288 288 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 152 152 152 306 306 306 F F F 9 180-199 146 146 146 291 291 291 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 148 148 148 302 302 302 F F F 12 240-259 152 152 152 306 306 306 F F F 13 260-279 149 149 149 303 303 303 F F F 14 280-299 151 151 151 305 305 305 F F F 15 300-319 152 152 152 306 306 306 F F F 16 320-339 139 139 139 278 278 278 F F F 17 340-359 152 152 152 306 306 306 F F F 18 360-379 149 149 149 303 303 303 F F F 19 380-399 153 153 153 308 308 308 F F F 20 400-419 152 152 152 307 307 307 F F F 21 420-439 152 152 152 306 306 306 F F F 22 440-459 148 148 148 298 298 298 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 144 144 144 287 287 287 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 152 152 152 306 306 306 F F F Max gen: 153, Max flush: 308 Begin loop 77 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.482633699Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.482918321Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.482935229Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801} {"msg":"Crucible c521efa7-7ee3-4751-8593-8f82daeb1960 has session id: e2705bb9-2db9-44a2-964d-9597e210347b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483350978Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Upstairs opts: Upstairs UUID: c521efa7-7ee3-4751-8593-8f82daeb1960, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483385395Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Crucible stats registered with UUID: c521efa7-7ee3-4751-8593-8f82daeb1960","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483404353Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483789675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801} {"msg":"c521efa7-7ee3-4751-8593-8f82daeb1960 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483814582Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483829181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"0","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483843699Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"0","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483857718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"1","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483880526Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"1","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483903453Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483927061Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483951059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"io task","client":"0","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.483975296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"io task","client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.484017752Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"io task","client":"1","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"ds_connection connected from Ok(127.0.0.1:45379)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.484150599Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"io task","client":"0","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"ds_connection connected from Ok(127.0.0.1:54698)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.48423659Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"io task","client":"1","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:53:46.484448839Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"io task","client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:53:46.484491325Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"io task","client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:53:46.484583676Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.484601584Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.484615823Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.484713023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"io task","client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.485109054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"0","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.485135291Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"0","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.48515042Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"1","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.485164138Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"1","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.485714004Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"0","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.485826983Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"1","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.486249961Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"0","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.486286877Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"0","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.486314514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.486344621Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"1","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.486372849Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"1","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:46.486394606Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.485322447Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"io task","client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"ds_connection connected from Ok(127.0.0.1:47951)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.48549814Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"io task","client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.48650956Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.486545716Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487137048Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487678894Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487700932Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[0]R flush_numbers[0..12]: [305, 304, 304, 305, 303, 288, 264, 276, 306, 291, 302, 302]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.48771579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[0]R generation[0..12]: [151, 150, 150, 151, 149, 144, 135, 138, 152, 146, 148, 148]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487730439Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487745098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[1]R flush_numbers[0..12]: [305, 304, 304, 305, 303, 288, 264, 276, 306, 291, 302, 302]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487759316Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[1]R generation[0..12]: [151, 150, 150, 151, 149, 144, 135, 138, 152, 146, 148, 148]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487773185Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487791023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[2]R flush_numbers[0..12]: [305, 304, 304, 305, 303, 288, 264, 276, 306, 291, 302, 302]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487805341Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[2]R generation[0..12]: [151, 150, 150, 151, 149, 144, 135, 138, 152, 146, 148, 148]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.48781924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487833279Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Max found gen is 154","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487846407Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Generation requested: 154 >= found:154","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2024-07-09T21:53:56.487862716Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Next flush: 309","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487876154Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487889123Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"":"downstairs","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487902242Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487914331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"0","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487927759Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"1","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487940908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"client":"2","":"downstairs","session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487954117Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487966565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"c521efa7-7ee3-4751-8593-8f82daeb1960 is now active with session: e2705bb9-2db9-44a2-964d-9597e210347b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487979624Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.487993883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801,"session_id":"e2705bb9-2db9-44a2-964d-9597e210347b"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:53:56.488006721Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 536 len 16384 data: 10 10 13 12 02/30 Read block 502 len 4096 03/30 Flush 04/30 Write block 345 len 24576 data: 12 13 10 9 9 10 05/30 Read block 113 len 24576 06/30 Read block 525 len 24576 07/30 Read block 285 len 16384 08/30 Write block 113 len 20480 data: 15 17 15 12 12 09/30 Read block 248 len 28672 10/30 Write block 385 len 36864 data: 19 21 21 21 21 21 21 20 17 11/30 Read block 210 len 20480 12/30 Write block 100 len 16384 data: 14 13 12 16 13/30 Read block 526 len 32768 14/30 Read block 318 len 24576 15/30 Write block 203 len 16384 data: 13 16 19 20 16/30 Read block 139 len 24576 17/30 Write block 366 len 20480 data: 13 11 11 16 13 18/30 Write block 218 len 24576 data: 13 14 10 12 14 13 19/30 Write block 365 len 24576 data: 12 14 12 12 17 14 20/30 Read block 346 len 16384 21/30 Write block 220 len 36864 data: 11 13 15 14 13 12 10 6 5 22/30 Read block 331 len 20480 23/30 Flush 24/30 Write block 189 len 12288 data: 13 13 16 25/30 Read block 284 len 16384 26/30 Read block 415 len 36864 27/30 Read block 474 len 12288 28/30 Write block 321 len 16384 data: 15 14 13 11 29/30 Read block 494 len 4096 30/30 Read block 292 len 8192 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 3 Done Done Done false 26 Acked 1025 Read 4 Done Done Done false 27 Acked 1026 Read 9 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Write 4 Done Done Done false 30 Acked 1029 Read 1 Done Done Done false 31 Acked 1030 Read 2 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 7 6 6 19 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:53:56.51556367Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1801} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 150 150 150 304 304 304 F F F 3 060-079 151 151 151 305 305 305 F F F 4 080-099 149 149 149 303 303 303 F F F 5 100-119 154 154 154 310 310 310 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 152 152 152 306 306 306 F F F 9 180-199 146 146 146 291 291 291 T T T 10 200-219 154 154 154 310 310 310 F F F 11 220-239 154 154 154 311 311 311 F F F 12 240-259 152 152 152 306 306 306 F F F 13 260-279 149 149 149 303 303 303 F F F 14 280-299 151 151 151 305 305 305 F F F 15 300-319 152 152 152 306 306 306 F F F 16 320-339 139 139 139 278 278 278 T T T 17 340-359 154 154 154 310 310 310 F F F 18 360-379 154 154 154 310 310 310 F F F 19 380-399 154 154 154 310 310 310 F F F 20 400-419 152 152 152 307 307 307 F F F 21 420-439 152 152 152 306 306 306 F F F 22 440-459 148 148 148 298 298 298 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 154 154 154 309 309 309 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 152 152 152 306 306 306 F F F Max gen: 154, Max flush: 311 A Difference in extent metadata is expected here On loop 77 Verifying data now No change after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 150 150 150 304 304 304 F F F 3 060-079 151 151 151 305 305 305 F F F 4 080-099 149 149 149 303 303 303 F F F 5 100-119 154 154 154 310 310 310 F F F 6 120-139 135 135 135 264 264 264 F F F 7 140-159 138 138 138 276 276 276 F F F 8 160-179 152 152 152 306 306 306 F F F 9 180-199 155 155 155 312 312 312 F F F 10 200-219 154 154 154 310 310 310 F F F 11 220-239 154 154 154 311 311 311 F F F 12 240-259 152 152 152 306 306 306 F F F 13 260-279 149 149 149 303 303 303 F F F 14 280-299 151 151 151 305 305 305 F F F 15 300-319 152 152 152 306 306 306 F F F 16 320-339 155 155 155 312 312 312 F F F 17 340-359 154 154 154 310 310 310 F F F 18 360-379 154 154 154 310 310 310 F F F 19 380-399 154 154 154 310 310 310 F F F 20 400-419 152 152 152 307 307 307 F F F 21 420-439 152 152 152 306 306 306 F F F 22 440-459 148 148 148 298 298 298 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 133 133 133 258 258 258 F F F 26 520-539 154 154 154 309 309 309 F F F 27 540-559 144 144 144 288 288 288 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 152 152 152 306 306 306 F F F Max gen: 155, Max flush: 312 Begin loop 78 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.70236185Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.702786338Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.702870739Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809} {"msg":"Crucible 365f52d1-334d-40ff-8978-ea3befbf7741 has session id: 167281b0-5eb8-45e5-a629-b641d46cc94b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703242093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Upstairs opts: Upstairs UUID: 365f52d1-334d-40ff-8978-ea3befbf7741, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703276299Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Crucible stats registered with UUID: 365f52d1-334d-40ff-8978-ea3befbf7741","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703292928Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.70367589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809} {"msg":"365f52d1-334d-40ff-8978-ea3befbf7741 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703715506Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703731964Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703752282Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703777069Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"1","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703803257Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"1","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703826675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"2","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703850992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"2","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.70387324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"io task","client":"2","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703897028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"io task","client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.703920785Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"io task","client":"1","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"ds_connection connected from Ok(127.0.0.1:35742)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.70407646Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"io task","client":"2","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"ds_connection connected from Ok(127.0.0.1:52483)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.704256692Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"io task","client":"1","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:10.704300608Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"io task","client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:10.704328035Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"io task","client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:10.704354152Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.70437402Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.704395748Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.704446363Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"io task","client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.70497789Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"1","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.705005288Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"1","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.705020196Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"2","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.705034065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"2","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.705668202Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"1","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.70569539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"2","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.706278512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"1","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.706305779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"1","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.706321627Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.706338426Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"2","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.706353454Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"2","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:10.706367043Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.70473331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"io task","client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"ds_connection connected from Ok(127.0.0.1:36084)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.704850958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"io task","client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.705759808Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.705797364Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.706395235Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.706914784Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.706938051Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[0]R flush_numbers[0..12]: [305, 304, 304, 305, 303, 310, 264, 276, 306, 312, 310, 311]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.70695381Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[0]R generation[0..12]: [151, 150, 150, 151, 149, 154, 135, 138, 152, 155, 154, 154]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.706968518Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.706983257Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[1]R flush_numbers[0..12]: [305, 304, 304, 305, 303, 310, 264, 276, 306, 312, 310, 311]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.706998125Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[1]R generation[0..12]: [151, 150, 150, 151, 149, 154, 135, 138, 152, 155, 154, 154]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707015814Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707030292Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[2]R flush_numbers[0..12]: [305, 304, 304, 305, 303, 310, 264, 276, 306, 312, 310, 311]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707044431Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[2]R generation[0..12]: [151, 150, 150, 151, 149, 154, 135, 138, 152, 155, 154, 154]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707058389Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707072798Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707085787Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Generation requested: 156 >= found:156","v":0,"name":"crucible",Wait for a query_work_queue command to finish before sending IO "level":30,"time":"2024-07-09T21:54:20.707103115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Next flush: 313","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707116533Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707129512Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"":"downstairs","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707143441Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.70715663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"0","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707170768Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"1","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707184407Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"client":"2","":"downstairs","session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707197985Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707210424Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"365f52d1-334d-40ff-8978-ea3befbf7741 is now active with session: 167281b0-5eb8-45e5-a629-b641d46cc94b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707225583Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.707238661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809,"session_id":"167281b0-5eb8-45e5-a629-b641d46cc94b"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:20.70725101Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 53 len 12288 data: 14 14 14 02/30 Write block 580 len 32768 data: 9 10 11 9 11 10 11 15 03/30 Read block 245 len 40960 04/30 Write block 555 len 4096 data: 17 05/30 Write block 131 len 16384 data: 8 8 10 11 06/30 Read block 400 len 40960 07/30 Read block 153 len 16384 08/30 Read block 335 len 16384 09/30 Read block 485 len 40960 10/30 Write block 502 len 8192 data: 6 7 11/30 Read block 326 len 16384 12/30 Read block 208 len 32768 13/30 Read block 310 len 28672 14/30 Read block 46 len 36864 15/30 Read block 64 len 28672 16/30 Write block 265 len 32768 data: 12 11 10 6 8 6 8 7 17/30 Write block 521 len 4096 data: 6 18/30 Write block 41 len 8192 data: 10 10 19/30 Read block 178 len 12288 20/30 Write block 158 len 36864 data: 7 7 11 9 10 11 11 14 12 21/30 Read block 512 len 12288 22/30 Write block 119 len 28672 data: 9 8 8 10 14 16 16 23/30 Write block 531 len 4096 data: 11 24/30 Read block 93 len 20480 25/30 Read block 69 len 40960 26/30 Read block 456 len 40960 27/30 Read block 471 len 32768 28/30 Write block 407 len 36864 data: 16 18 17 14 12 11 9 9 10 29/30 Read block 478 len 12288 30/30 Write block 452 len 16384 data: 16 15 16 15 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Read 10 Done Done Done false 28 Acked 1027 Read 8 Done Done Done false 29 Acked 1028 Write 9 Done Done Done false 30 Acked 1029 Read 3 Done Done Done false 31 Acked 1030 Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:20.732488979Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1809} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 156 156 156 313 313 313 F F F 3 060-079 151 151 151 305 305 305 F F F 4 080-099 149 149 149 303 303 303 F F F 5 100-119 156 156 156 313 313 313 F F F 6 120-139 156 156 156 313 313 313 F F F 7 140-159 156 156 156 313 313 313 F F F 8 160-179 156 156 156 313 313 313 F F F 9 180-199 155 155 155 312 312 312 F F F 10 200-219 154 154 154 310 310 310 F F F 11 220-239 154 154 154 311 311 311 F F F 12 240-259 152 152 152 306 306 306 F F F 13 260-279 156 156 156 313 313 313 F F F 14 280-299 151 151 151 305 305 305 F F F 15 300-319 152 152 152 306 306 306 F F F 16 320-339 155 155 155 312 312 312 F F F 17 340-359 154 154 154 310 310 310 F F F 18 360-379 154 154 154 310 310 310 F F F 19 380-399 154 154 154 310 310 310 F F F 20 400-419 152 152 152 307 307 307 T T T 21 420-439 152 152 152 306 306 306 F F F 22 440-459 148 148 148 298 298 298 T T T 23 460-479 138 138 138 275 275 275 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 156 156 156 313 313 313 F F F 26 520-539 156 156 156 313 313 313 F F F 27 540-559 156 156 156 313 313 313 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 156, Max flush: 313 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 156 156 156 313 313 313 F F F 3 060-079 151 151 151 305 305 305 F F F 4 080-099 149 149 149 303 303 303 F F F 5 100-119 156 156 156 313 313 313 F F F 6 120-139 156 156 156 313 313 313 F F F 7 140-159 156 156 156 313 313 313 F F F 8 160-179 156 156 156 313 313 313 F F F 9 180-199 155 155 155 312 312 312 F F F 10 200-219 154 154 154 310 310 310 F F F 11 220-239 154 154 154 311 311 311 F F F 12 240-259 152 152 152 306 306 306 F F F 13 260-279 156 156 156 313 313 313 F F F 14 280-299 151 151 151 305 305 305 F F F 15 300-319 152 152 152 306 306 306 F F F 16 320-339 155 155 155 312 312 312 F F F 17 340-359 154 154 154 310 310 310 F F F 18 360-379 154 154 154 310 310 310 F F F 19 380-399 154 154 154 310 310 310 F F F 20 400-419 157 157 157 314 314 314 F F F 21 420-439 152 152 152 306 306 306 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 138 138 138 275 275 275 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 156 156 156 313 313 313 F F F 26 520-539 156 156 156 313 313 313 F F F 27 540-559 156 156 156 313 313 313 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 157, Max flush: 314 Begin loop 79 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.918386187Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.918699976Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.918717714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817} {"msg":"Crucible 2719bf57-5cda-44aa-9c54-abdc7b479ca0 has session id: eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919188307Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Upstairs opts: Upstairs UUID: 2719bf57-5cda-44aa-9c54-abdc7b479ca0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919236493Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Crucible stats registered with UUID: 2719bf57-5cda-44aa-9c54-abdc7b479ca0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.91926107Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919559151Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817} {"msg":"2719bf57-5cda-44aa-9c54-abdc7b479ca0 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919596617Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919615375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919630834Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919644732Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"1","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919658271Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"1","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919672179Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"2","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919685698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"2","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919759701Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"io task","client":"1","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919781929Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"io task","client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.919796337Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"io task","client":"2","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:34.919953302Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"io task","client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:34.919994038Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"io task","client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"ds_connection connected from Ok(127.0.0.1:61406)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.920024464Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"io task","client":"1","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:34.920053812Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.920076079Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.920137953Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.920163431Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"io task","client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"ds_connection connected from Ok(127.0.0.1:50191)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.920251552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"io task","client":"2","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.920754982Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"1","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.920784309Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"1","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.920805807Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"2","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.920826925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"2","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.921341344Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"2","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.921469951Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"1","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.921809118Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"2","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.921846334Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"2","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.921878891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.921905638Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"1","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.921933645Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"1","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:34.921955393Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.921097136Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"io task","client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"ds_connection connected from Ok(127.0.0.1:39956)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.921218714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"io task","client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.922629794Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.922657321Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.92317187Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923772251Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923796289Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[0]R flush_numbers[0..12]: [305, 304, 313, 305, 303, 313, 313, 313, 313, 312, 310, 311]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923814677Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[0]R generation[0..12]: [151, 150, 156, 151, 149, 156, 156, 156, 156, 155, 154, 154]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923832265Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923850513Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[1]R flush_numbers[0..12]: [305, 304, 313, 305, 303, 313, 313, 313, 313, 312, 310, 311]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923868192Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[1]R generation[0..12]: [151, 150, 156, 151, 149, 156, 156, 156, 156, 155, 154, 154]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923889669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923908017Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[2]R flush_numbers[0..12]: [305, 304, 313, 305, 303, 313, 313, 313, 313, 312, 310, 311]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923925646Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[2]R generation[0..12]: [151, 150, 156, 151, 149, 156, 156, 156, 156, 155, 154, 154]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:54:44.923943064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923964532Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Max found gen is 158","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.92398102Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.923998728Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Next flush: 315","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.924026346Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.924055943Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"":"downstairs","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.92408635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.924109697Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"0","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.924135695Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"1","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.924162392Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"client":"2","":"downstairs","session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.92418924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.924214197Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"2719bf57-5cda-44aa-9c54-abdc7b479ca0 is now active with session: eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.924243154Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.924267882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817,"session_id":"eb68a4c6-5ac1-4fa5-8bfb-a89dd500fb2b"} {"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:44.924291379Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 } Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 393 len 8192 data: 18 19 02/30 Read block 393 len 36864 03/30 Read block 55 len 16384 04/30 Read block 10 len 32768 05/30 Read block 246 len 20480 06/30 Read block 268 len 4096 07/30 Write block 358 len 20480 data: 18 16 15 12 14 08/30 Read block 47 len 28672 09/30 Write block 107 len 8192 data: 12 13 10/30 Read block 406 len 20480 11/30 Write block 469 len 16384 data: 11 10 7 8 12/30 Write block 250 len 32768 data: 5 7 7 7 6 9 10 9 13/30 Flush 14/30 Write block 295 len 28672 data: 9 8 8 10 10 13 12 15/30 Read block 9 len 16384 16/30 Read block 29 len 12288 17/30 Read block 230 len 32768 18/30 Write block 79 len 28672 data: 9 11 12 13 13 14 14 19/30 Write block 82 len 28672 data: 14 14 15 15 16 15 15 20/30 Read block 270 len 28672 21/30 Read block 142 len 16384 22/30 Read block 455 len 28672 23/30 Read block 326 len 28672 24/30 Read block 304 len 32768 25/30 Write block 241 len 8192 data: 11 11 26/30 Write block 175 len 16384 data: 11 9 12 10 27/30 Read block 362 len 4096 28/30 Read block 548 len 36864 29/30 Write block 393 len 12288 data: 19 20 17 30/30 Read block 236 len 32768 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Read 8 Done Done Done false 19 Acked 1018 Write 7 Done Done Done false 20 Acked 1019 Write 7 Done Done Done false 21 Acked 1020 Read 7 Done Done Done false 22 Acked 1021 Read 4 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Read 7 Done Done Done false 25 Acked 1024 Read 8 Done Done Done false 26 Acked 1025 Write 2 Done Done Done false 27 Acked 1026 Write 4 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Write 3 Done Done Done false 31 Acked 1030 Read 8 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 14 14 14 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:44.950543628Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1817} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 156 156 156 313 313 313 F F F 3 060-079 151 151 151 305 305 305 T T T 4 080-099 149 149 149 303 303 303 T T T 5 100-119 158 158 158 315 315 315 F F F 6 120-139 156 156 156 313 313 313 F F F 7 140-159 156 156 156 313 313 313 F F F 8 160-179 156 156 156 313 313 313 T T T 9 180-199 155 155 155 312 312 312 F F F 10 200-219 154 154 154 310 310 310 F F F 11 220-239 154 154 154 311 311 311 F F F 12 240-259 158 158 158 315 315 315 T T T 13 260-279 156 156 156 313 313 313 F F F 14 280-299 158 158 158 316 316 316 F F F 15 300-319 158 158 158 316 316 316 F F F 16 320-339 155 155 155 312 312 312 F F F 17 340-359 158 158 158 315 315 315 F F F 18 360-379 158 158 158 315 315 315 F F F 19 380-399 158 158 158 315 315 315 T T T 20 400-419 157 157 157 314 314 314 F F F 21 420-439 152 152 152 306 306 306 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 156 156 156 313 313 313 F F F 26 520-539 156 156 156 313 313 313 F F F 27 540-559 156 156 156 313 313 313 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 158, Max flush: 316 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 156 156 156 313 313 313 F F F 3 060-079 159 159 159 317 317 317 F F F 4 080-099 159 159 159 317 317 317 F F F 5 100-119 158 158 158 315 315 315 F F F 6 120-139 156 156 156 313 313 313 F F F 7 140-159 156 156 156 313 313 313 F F F 8 160-179 159 159 159 317 317 317 F F F 9 180-199 155 155 155 312 312 312 F F F 10 200-219 154 154 154 310 310 310 F F F 11 220-239 154 154 154 311 311 311 F F F 12 240-259 159 159 159 317 317 317 F F F 13 260-279 156 156 156 313 313 313 F F F 14 280-299 158 158 158 316 316 316 F F F 15 300-319 158 158 158 316 316 316 F F F 16 320-339 155 155 155 312 312 312 F F F 17 340-359 158 158 158 315 315 315 F F F 18 360-379 158 158 158 315 315 315 F F F 19 380-399 159 159 159 317 317 317 F F F 20 400-419 157 157 157 314 314 314 F F F 21 420-439 152 152 152 306 306 306 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 156 156 156 313 313 313 F F F 26 520-539 156 156 156 313 313 313 F F F 27 540-559 156 156 156 313 313 313 F F F 28 560-579 143 143 143 286 286 286 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 159, Max flush: 317 Begin loop 80 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.181239628Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.181772366Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.181807662Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825} {"msg":"Crucible 23c3d62d-f03e-4890-ba65-27651d606447 has session id: a087b665-f3bc-494f-b788-8ff4b825033c","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182054987Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Upstairs opts: Upstairs UUID: 23c3d62d-f03e-4890-ba65-27651d606447, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182099633Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Crucible stats registered with UUID: 23c3d62d-f03e-4890-ba65-27651d606447","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.18212396Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182516722Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825} {"msg":"23c3d62d-f03e-4890-ba65-27651d606447 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182544729Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182558628Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182573176Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182586805Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"1","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182600563Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"1","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182613802Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"2","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182627511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"2","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182688245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"io task","client":"1","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182707853Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"io task","client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.182728131Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"io task","client":"2","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:59.18293338Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"io task","client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:59.182982066Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"io task","client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:54:59.183110753Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.18314152Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.183164337Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"ds_connection connected from Ok(127.0.0.1:64723)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.183189715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"io task","client":"1","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.183233801Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"io task","client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"ds_connection connected from Ok(127.0.0.1:38431)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.183264658Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"io task","client":"2","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.183698855Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"1","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.183726412Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"1","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.18374127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"2","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.183755889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"2","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.184303675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"2","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.184440031Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"1","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.184778968Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"2","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.184807425Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"2","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.184823663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.184863799Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"1","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.184881207Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"1","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:54:59.184895476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.184262855Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"io task","client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"ds_connection connected from Ok(127.0.0.1:63876)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.184473304Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"io task","client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.185572775Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.185603772Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186165086Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186693364Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186719181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[0]R flush_numbers[0..12]: [305, 304, 313, 317, 317, 315, 313, 313, 317, 312, 310, 311]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.18673807Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[0]R generation[0..12]: [151, 150, 156, 159, 159, 158, 156, 156, 159, 155, 154, 154]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186757198Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186775486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[1]R flush_numbers[0..12]: [305, 304, 313, 317, 317, 315, 313, 313, 317, 312, 310, 311]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186799213Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[1]R generation[0..12]: [151, 150, 156, 159, 159, 158, 156, 156, 159, 155, 154, 154]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186817332Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.1868354Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[2]R flush_numbers[0..12]: [305, 304, 313, 317, 317, 315, 313, 313, 317, 312, 310, 311]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:55:09.186854448Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[2]R generation[0..12]: [151, 150, 156, 159, 159, 158, 156, 156, 159, 155, 154, 154]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186875786Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186893694Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186910093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.18692978Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Next flush: 318","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186953858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186977696Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"":"downstairs","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.186998104Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.187016212Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"0","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.18703906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"1","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.187062837Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"client":"2","":"downstairs","session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.187090245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.187112122Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"23c3d62d-f03e-4890-ba65-27651d606447 is now active with session: a087b665-f3bc-494f-b788-8ff4b825033c","v":0,"name":"crucible"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"level":30,"time":"2024-07-09T21:55:09.18713872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.187160388Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825,"session_id":"a087b665-f3bc-494f-b788-8ff4b825033c"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:09.187182155Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 400 len 20480 data: 11 8 7 12 12 02/30 Read block 324 len 8192 03/30 Read block 164 len 36864 04/30 Read block 317 len 8192 05/30 Write block 44 len 40960 data: 14 13 11 11 10 9 6 9 9 15 06/30 Read block 565 len 24576 07/30 Read block 11 len 24576 08/30 Read block 280 len 12288 09/30 Read block 101 len 28672 10/30 Read block 63 len 28672 11/30 Write block 520 len 4096 data: 9 12/30 Write block 297 len 40960 data: 9 11 11 14 13 10 9 12 14 14 13/30 Write block 571 len 12288 data: 21 18 18 14/30 Read block 306 len 8192 15/30 Read block 581 len 32768 16/30 Flush 17/30 Write block 215 len 36864 data: 13 12 17 14 15 12 14 16 15 18/30 Read block 447 len 28672 19/30 Write block 419 len 28672 data: 10 13 12 13 11 10 11 20/30 Read block 12 len 4096 21/30 Read block 69 len 36864 22/30 Flush 23/30 Flush 24/30 Write block 149 len 36864 data: 11 10 11 9 8 9 9 8 6 25/30 Read block 220 len 40960 26/30 Read block 533 len 28672 27/30 Read block 34 len 16384 28/30 Write block 215 len 24576 data: 14 13 18 15 16 13 29/30 Read block 61 len 28672 30/30 Read block 472 len 28672 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 9 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Read 4 Done Done Done false 29 Acked 1028 Write 6 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 7 6 6 19 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:55:09.215035965Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1825} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 160 160 160 318 318 318 F F F 3 060-079 159 159 159 317 317 317 F F F 4 080-099 159 159 159 317 317 317 F F F 5 100-119 158 158 158 315 315 315 F F F 6 120-139 156 156 156 313 313 313 F F F 7 140-159 156 156 156 313 313 313 T T T 8 160-179 159 159 159 317 317 317 F F F 9 180-199 155 155 155 312 312 312 F F F 10 200-219 160 160 160 319 319 319 T T T 11 220-239 160 160 160 319 319 319 T T T 12 240-259 159 159 159 317 317 317 F F F 13 260-279 156 156 156 313 313 313 F F F 14 280-299 160 160 160 318 318 318 F F F 15 300-319 160 160 160 318 318 318 F F F 16 320-339 155 155 155 312 312 312 F F F 17 340-359 158 158 158 315 315 315 F F F 18 360-379 158 158 158 315 315 315 F F F 19 380-399 159 159 159 317 317 317 F F F 20 400-419 160 160 160 319 319 319 F F F 21 420-439 160 160 160 319 319 319 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 156 156 156 313 313 313 F F F 26 520-539 160 160 160 318 318 318 F F F 27 540-559 156 156 156 313 313 313 F F F 28 560-579 160 160 160 318 318 318 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 160, Max flush: 319 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 150 150 150 304 304 304 F F F 2 040-059 160 160 160 318 318 318 F F F 3 060-079 159 159 159 317 317 317 F F F 4 080-099 159 159 159 317 317 317 F F F 5 100-119 158 158 158 315 315 315 F F F 6 120-139 156 156 156 313 313 313 F F F 7 140-159 161 161 161 320 320 320 F F F 8 160-179 159 159 159 317 317 317 F F F 9 180-199 155 155 155 312 312 312 F F F 10 200-219 161 161 161 320 320 320 F F F 11 220-239 161 161 161 320 320 320 F F F 12 240-259 159 159 159 317 317 317 F F F 13 260-279 156 156 156 313 313 313 F F F 14 280-299 160 160 160 318 318 318 F F F 15 300-319 160 160 160 318 318 318 F F F 16 320-339 155 155 155 312 312 312 F F F 17 340-359 158 158 158 315 315 315 F F F 18 360-379 158 158 158 315 315 315 F F F 19 380-399 159 159 159 317 317 317 F F F 20 400-419 160 160 160 319 319 319 F F F 21 420-439 160 160 160 319 319 319 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 156 156 156 313 313 313 F F F 26 520-539 160 160 160 318 318 318 F F F 27 540-559 156 156 156 313 313 313 F F F 28 560-579 160 160 160 318 318 318 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 161, Max flush: 320 Begin loop 81 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.414267375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.414728629Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.414761946Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833} {"msg":"Crucible b18432fa-bc3a-48f7-b4ac-060b1c301ea0 has session id: 85baf744-7351-4111-aa87-054ea53e6c61","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.414788093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Upstairs opts: Upstairs UUID: b18432fa-bc3a-48f7-b4ac-060b1c301ea0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.414932149Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Crucible stats registered with UUID: b18432fa-bc3a-48f7-b4ac-060b1c301ea0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.414965496Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415256637Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833} {"msg":"b18432fa-bc3a-48f7-b4ac-060b1c301ea0 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.41532668Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415346578Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"0","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415361787Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"0","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415375495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415390054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415420451Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"2","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415444938Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"2","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415460117Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"io task","client":"0","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415474925Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"io task","client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415540769Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"io task","client":"2","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"ds_connection connected from Ok(127.0.0.1:42071)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415760057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"io task","client":"2","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:55:23.415868556Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"io task","client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"ds_connection connected from Ok(127.0.0.1:65210)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415901593Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"io task","client":"0","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:55:23.415922171Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"io task","client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:55:23.415939459Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415952068Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.415969756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.416026921Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"io task","client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.41643203Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"2","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.416463617Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"2","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.416488975Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"0","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.416515232Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"0","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.416949399Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"0","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.417061618Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"2","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.417391805Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"0","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.417414003Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"0","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.417427872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.41755053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"2","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.417568528Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"2","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:23.417582047Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.416735445Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"io task","client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"ds_connection connected from Ok(127.0.0.1:34005)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.416999699Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"io task","client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.418356055Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.41840437Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.418934877Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.41951732Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419537698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[0]R flush_numbers[0..12]: [305, 304, 318, 317, 317, 315, 313, 320, 317, 312, 320, 320]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419552966Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[0]R generation[0..12]: [151, 150, 160, 159, 159, 158, 156, 161, 159, 155, 161, 161]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419567665Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419582663Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[1]R flush_numbers[0..12]: [305, 304, 318, 317, 317, 315, 313, 320, 317, 312, 320, 320]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419600411Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[1]R generation[0..12]: [151, 150, 160, 159, 159, 158, 156, 161, 159, 155, 161, 161]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.41961501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419629339Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[2]R flush_numbers[0..12]: [305, 304, 318, 317, 317, 315, 313, 320, 317, 312, 320, 320]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419643437Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[2]R generation[0..12]: [151, 150, 160, 159, 159, 158, 156, 161, 159, 155, 161, 161]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419657346Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419671335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Max found gen is 162","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419684313Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419697682Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Next flush: 321","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:55:33.41971066Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419725179Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"":"downstairs","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419738528Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419750476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"0","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419764785Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"1","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419777994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"client":"2","":"downstairs","session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419793522Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419806031Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"b18432fa-bc3a-48f7-b4ac-060b1c301ea0 is now active with session: 85baf744-7351-4111-aa87-054ea53e6c61","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.41981907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419831139Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833,"session_id":"85baf744-7351-4111-aa87-054ea53e6c61"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:33.419843457Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 291 len 28672 data: 11 10 9 10 10 9 10 02/30 Read block 506 len 40960 03/30 Read block 310 len 24576 04/30 Write block 282 len 20480 data: 9 10 11 12 10 05/30 Read block 297 len 4096 06/30 Write block 39 len 12288 data: 12 11 11 07/30 Read block 292 len 24576 08/30 Read block 572 len 36864 09/30 Read block 57 len 28672 10/30 Write block 203 len 8192 data: 14 17 11/30 Read block 432 len 28672 12/30 Read block 433 len 28672 13/30 Read block 207 len 24576 14/30 Write block 283 len 8192 data: 11 12 15/30 Read block 48 len 8192 16/30 Read block 161 len 36864 17/30 Read block 380 len 12288 18/30 Write block 260 len 40960 data: 9 6 9 12 10 13 12 11 7 9 19/30 Read block 94 len 24576 20/30 Read block 170 len 40960 21/30 Read block 388 len 20480 22/30 Write block 543 len 32768 data: 11 11 11 14 15 15 15 16 23/30 Read block 523 len 8192 24/30 Flush 25/30 Write block 391 len 16384 data: 22 21 20 21 26/30 Write block 268 len 40960 data: 8 10 7 9 8 8 7 9 8 8 27/30 Write block 303 len 8192 data: 10 13 28/30 Read block 34 len 4096 29/30 Read block 220 len 20480 30/30 Write block 54 len 4096 data: 15 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 4 Done Done Done false 26 Acked 1025 Write 10 Done Done Done false 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:55:33.444114332Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1833} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 162 162 162 321 321 321 F F F 2 040-059 162 162 162 321 321 321 T T T 3 060-079 159 159 159 317 317 317 F F F 4 080-099 159 159 159 317 317 317 F F F 5 100-119 158 158 158 315 315 315 F F F 6 120-139 156 156 156 313 313 313 F F F 7 140-159 161 161 161 320 320 320 F F F 8 160-179 159 159 159 317 317 317 F F F 9 180-199 155 155 155 312 312 312 F F F 10 200-219 162 162 162 321 321 321 F F F 11 220-239 161 161 161 320 320 320 F F F 12 240-259 159 159 159 317 317 317 F F F 13 260-279 162 162 162 321 321 321 T T T 14 280-299 162 162 162 321 321 321 F F F 15 300-319 160 160 160 318 318 318 T T T 16 320-339 155 155 155 312 312 312 F F F 17 340-359 158 158 158 315 315 315 F F F 18 360-379 158 158 158 315 315 315 F F F 19 380-399 159 159 159 317 317 317 T T T 20 400-419 160 160 160 319 319 319 F F F 21 420-439 160 160 160 319 319 319 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 156 156 156 313 313 313 F F F 26 520-539 160 160 160 318 318 318 F F F 27 540-559 162 162 162 321 321 321 F F F 28 560-579 160 160 160 318 318 318 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 162, Max flush: 321 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 162 162 162 321 321 321 F F F 2 040-059 163 163 163 322 322 322 F F F 3 060-079 159 159 159 317 317 317 F F F 4 080-099 159 159 159 317 317 317 F F F 5 100-119 158 158 158 315 315 315 F F F 6 120-139 156 156 156 313 313 313 F F F 7 140-159 161 161 161 320 320 320 F F F 8 160-179 159 159 159 317 317 317 F F F 9 180-199 155 155 155 312 312 312 F F F 10 200-219 162 162 162 321 321 321 F F F 11 220-239 161 161 161 320 320 320 F F F 12 240-259 159 159 159 317 317 317 F F F 13 260-279 163 163 163 322 322 322 F F F 14 280-299 162 162 162 321 321 321 F F F 15 300-319 163 163 163 322 322 322 F F F 16 320-339 155 155 155 312 312 312 F F F 17 340-359 158 158 158 315 315 315 F F F 18 360-379 158 158 158 315 315 315 F F F 19 380-399 163 163 163 322 322 322 F F F 20 400-419 160 160 160 319 319 319 F F F 21 420-439 160 160 160 319 319 319 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 156 156 156 313 313 313 F F F 26 520-539 160 160 160 318 318 318 F F F 27 540-559 162 162 162 321 321 321 F F F 28 560-579 160 160 160 318 318 318 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 163, Max flush: 322 Begin loop 82 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.658617206Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.65897867Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.659009707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841} {"msg":"Crucible a6f8e8c1-66c2-4f2b-a331-c4b30e8d3576 has session id: a01b2408-324e-472e-bc29-93ba1d6f4ba8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.659511527Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Upstairs opts: Upstairs UUID: a6f8e8c1-66c2-4f2b-a331-c4b30e8d3576, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.659555833Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Crucible stats registered with UUID: a6f8e8c1-66c2-4f2b-a331-c4b30e8d3576","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.65958074Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.65988152Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841} {"msg":"a6f8e8c1-66c2-4f2b-a331-c4b30e8d3576 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.659956183Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.659978971Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"0","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660003948Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"0","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660028806Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660052093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660074511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"2","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660099779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"2","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660123116Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"io task","client":"0","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660146924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"io task","client":"2","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660172801Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"io task","client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"ds_connection connected from Ok(127.0.0.1:33797)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660279831Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"io task","client":"2","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:55:47.660317247Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"io task","client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:55:47.660345044Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"io task","client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:55:47.660374421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"ds_connection connected from Ok(127.0.0.1:35126)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660398419Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"io task","client":"0","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660424186Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660447824Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.660627646Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"io task","client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.661179721Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"2","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.661211128Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"2","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.661225727Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"0","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.661245165Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"0","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.661844286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"2","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.661872753Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"0","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.662421618Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"0","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.662449486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"0","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.662464774Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.662477573Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"2","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.662491762Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"2","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:47.66250514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.661574936Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"io task","client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"ds_connection connected from Ok(127.0.0.1:32927)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.661719732Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"io task","client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.663088046Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.663134462Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.663700556Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664242442Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.66426336Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[0]R flush_numbers[0..12]: [305, 321, 322, 317, 317, 315, 313, 320, 317, 312, 321, 320]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664278858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[0]R generation[0..12]: [151, 162, 163, 159, 159, 158, 156, 161, 159, 155, 162, 161]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664293477Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664312165Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[1]R flush_numbers[0..12]: [305, 321, 322, 317, 317, 315, 313, 320, 317, 312, 321, 320]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664326854Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[1]R generation[0..12]: [151, 162, 163, 159, 159, 158, 156, 161, 159, 155, 162, 161]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664340882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664355311Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[2]R flush_numbers[0..12]: [305, 321, 322, 317, 317, 315, 313, 320, 317, 312, 321, 320]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664369299Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[2]R generation[0..12]: [151, 162, 163, 159, 159, 158, 156, 161, 159, 155, 162, 161]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664383318Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664397636Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664410815Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664427724Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Next flush: 323","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664440812Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664453921Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"":"downstairs","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.6644676Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664479988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"0","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664493507Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"1","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664506806Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"client":"2","":"downstairs","session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664522204Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664534903Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"a6f8e8c1-66c2-4f2b-a331-c4b30e8d3576 is now active with session: a01b2408-324e-472e-bc29-93ba1d6f4ba8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664547862Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664559851Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841,"session_id":"a01b2408-324e-472e-bc29-93ba1d6f4ba8"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:55:57.664572209Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 283 len 40960 data: 12 13 13 11 8 11 13 12 12 11 02/30 Read block 265 len 40960 03/30 Read block 182 len 16384 04/30 Write block 108 len 36864 data: 14 13 11 14 14 16 18 16 13 05/30 Read block 464 len 28672 06/30 Flush 07/30 Flush 08/30 Write block 504 len 24576 data: 8 9 12 11 13 11 09/30 Read block 146 len 20480 10/30 Write block 343 len 36864 data: 12 13 13 14 11 10 10 11 10 11/30 Flush 12/30 Write block 21 len 16384 data: 13 10 9 10 13/30 Read block 159 len 24576 14/30 Read block 364 len 4096 15/30 Write block 514 len 8192 data: 11 9 16/30 Read block 393 len 8192 17/30 Flush 18/30 Write block 410 len 24576 data: 15 13 12 10 10 11 19/30 Read block 429 len 4096 20/30 Read block 232 len 28672 21/30 Read block 274 len 8192 22/30 Read block 185 len 20480 23/30 Write block 47 len 12288 data: 12 11 10 24/30 Read block 65 len 12288 25/30 Read block 394 len 32768 26/30 Read block 472 len 20480 27/30 Write block 192 len 32768 data: 15 15 15 13 10 12 13 13 28/30 Write block 514 len 24576 data: 12 10 8 11 9 8 29/30 Read block 114 len 12288 30/30 Write block 365 len 16384 data: 13 15 13 13 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 21 Acked 1020 Read 7 Done Done Done false 22 Acked 1021 Read 2 Done Done Done false 23 Acked 1022 Read 5 Done Done Done false 24 Acked 1023 Write 3 Done Done Done false 25 Acked 1024 Read 3 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Write 8 Done Done Done false 29 Acked 1028 Write 6 Done Done Done false 30 Acked 1029 Read 3 Done Done Done false 31 Acked 1030 Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 10 10 10 30 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1019 1019 1019 Downstairs last five completed: 1019 1018 1017 1016 1015 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:55:57.692097981Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1841} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 164 164 164 326 326 326 F F F 2 040-059 163 163 163 322 322 322 T T T 3 060-079 159 159 159 317 317 317 F F F 4 080-099 159 159 159 317 317 317 F F F 5 100-119 164 164 164 323 323 323 F F F 6 120-139 156 156 156 313 313 313 F F F 7 140-159 161 161 161 320 320 320 F F F 8 160-179 159 159 159 317 317 317 F F F 9 180-199 155 155 155 312 312 312 T T T 10 200-219 162 162 162 321 321 321 F F F 11 220-239 161 161 161 320 320 320 F F F 12 240-259 159 159 159 317 317 317 F F F 13 260-279 163 163 163 322 322 322 F F F 14 280-299 164 164 164 323 323 323 F F F 15 300-319 163 163 163 322 322 322 F F F 16 320-339 155 155 155 312 312 312 F F F 17 340-359 164 164 164 325 325 325 F F F 18 360-379 158 158 158 315 315 315 T T T 19 380-399 163 163 163 322 322 322 F F F 20 400-419 164 164 164 327 327 327 F F F 21 420-439 160 160 160 319 319 319 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 164 164 164 326 326 326 T T T 26 520-539 160 160 160 318 318 318 F F F 27 540-559 162 162 162 321 321 321 F F F 28 560-579 160 160 160 318 318 318 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 164, Max flush: 327 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 164 164 164 326 326 326 F F F 2 040-059 165 165 165 328 328 328 F F F 3 060-079 159 159 159 317 317 317 F F F 4 080-099 159 159 159 317 317 317 F F F 5 100-119 164 164 164 323 323 323 F F F 6 120-139 156 156 156 313 313 313 F F F 7 140-159 161 161 161 320 320 320 F F F 8 160-179 159 159 159 317 317 317 F F F 9 180-199 165 165 165 328 328 328 F F F 10 200-219 162 162 162 321 321 321 F F F 11 220-239 161 161 161 320 320 320 F F F 12 240-259 159 159 159 317 317 317 F F F 13 260-279 163 163 163 322 322 322 F F F 14 280-299 164 164 164 323 323 323 F F F 15 300-319 163 163 163 322 322 322 F F F 16 320-339 155 155 155 312 312 312 F F F 17 340-359 164 164 164 325 325 325 F F F 18 360-379 165 165 165 328 328 328 F F F 19 380-399 163 163 163 322 322 322 F F F 20 400-419 164 164 164 327 327 327 F F F 21 420-439 160 160 160 319 319 319 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 165 165 165 328 328 328 F F F 26 520-539 160 160 160 318 318 318 F F F 27 540-559 162 162 162 321 321 321 F F F 28 560-579 160 160 160 318 318 318 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 165, Max flush: 328 Begin loop 83 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.906021771Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.906422252Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.906453998Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849} {"msg":"Crucible 824ee13c-95a4-4785-8c5c-00da4b84b7fb has session id: e95df946-c5dc-4366-b49a-dc5b43e3516e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.906783216Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Upstairs opts: Upstairs UUID: 824ee13c-95a4-4785-8c5c-00da4b84b7fb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.906830211Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Crucible stats registered with UUID: 824ee13c-95a4-4785-8c5c-00da4b84b7fb","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.906852209Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907125442Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849} {"msg":"824ee13c-95a4-4785-8c5c-00da4b84b7fb active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907232021Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907250369Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"0","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907265718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"0","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907279556Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"1","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907293355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"1","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907307544Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907321262Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907334591Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"io task","client":"0","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.90734856Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"io task","client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907365358Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"io task","client":"1","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"ds_connection connected from Ok(127.0.0.1:36543)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907681737Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"io task","client":"1","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44690)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.907964429Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"io task","client":"0","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:56:11.908064909Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"io task","client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:56:11.908106514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"io task","client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:56:11.908137891Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.908162219Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.908186657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.908233872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"io task","client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.908511394Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"0","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.908544691Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"0","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.908571968Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"1","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.908598426Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"1","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.909155131Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"0","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.909192487Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"1","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.909757691Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"1","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.909787898Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"1","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.909813835Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.909829994Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"0","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.909844572Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"0","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:11.909857841Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.909363934Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"io task","client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"ds_connection connected from Ok(127.0.0.1:35573)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.909778893Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"io task","client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.910956986Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.910983363Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.911544438Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912062327Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912081685Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[0]R flush_numbers[0..12]: [305, 326, 328, 317, 317, 323, 313, 320, 317, 328, 321, 320]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912096413Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[0]R generation[0..12]: [151, 164, 165, 159, 159, 164, 156, 161, 159, 165, 162, 161]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912115481Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.91213024Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[1]R flush_numbers[0..12]: [305, 326, 328, 317, 317, 323, 313, 320, 317, 328, 321, 320]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912144458Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[1]R generation[0..12]: [151, 164, 165, 159, 159, 164, 156, 161, 159, 165, 162, 161]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912158437Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912172516Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[2]R flush_numbers[0..12]: [305, 326, 328, 317, 317, 323, 313, 320, 317, 328, 321, 320]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912186444Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[2]R generation[0..12]: [151, 164, 165, 159, 159, 164, 156, 161, 159, 165, 162, 161]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912200333Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912214521Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Max found gen is 166","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.91222744Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 166 >= found:166","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912244538Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Next flush: 329","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912257837Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912270886Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"":"downstairs","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912284145Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912296253Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"0","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912309592Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"1","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912324741Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"client":"2","":"downstairs","session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912338319Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912350628Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"824ee13c-95a4-4785-8c5c-00da4b84b7fb is now active with session: e95df946-c5dc-4366-b49a-dc5b43e3516e","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912363647Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912375716Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849,"session_id":"e95df946-c5dc-4366-b49a-dc5b43e3516e"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:21.912388064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 56 len 28672 data: 15 15 15 13 12 14 16 02/30 Write block 70 len 12288 data: 14 15 13 03/30 Read block 454 len 24576 04/30 Write block 520 len 8192 data: 10 7 05/30 Read block 386 len 32768 06/30 Read block 425 len 20480 07/30 Read block 287 len 20480 08/30 Read block 567 len 24576 09/30 Flush 10/30 Flush 11/30 Write block 138 len 36864 data: 12 12 12 11 12 12 12 12 13 12/30 Read block 156 len 20480 13/30 Read block 358 len 4096 14/30 Read block 493 len 8192 15/30 Flush 16/30 Write block 236 len 40960 data: 12 10 11 7 11 12 12 11 9 7 17/30 Read block 70 len 12288 18/30 Read block 575 len 20480 19/30 Read block 390 len 20480 20/30 Write block 357 len 24576 data: 15 19 17 16 13 15 21/30 Read block 326 len 4096 22/30 Read block 231 len 36864 23/30 Write block 316 len 24576 data: 13 16 18 16 15 16 24/30 Write block 399 len 16384 data: 11 12 9 8 25/30 Read block 40 len 20480 26/30 Read block 256 len 16384 27/30 Write block 212 len 28672 data: 16 15 14 15 14 19 16 28/30 Write block 575 len 8192 data: 14 17 29/30 Read block 453 len 20480 30/30 Write block 227 len 36864 data: 7 6 7 8 9 8 9 11 10 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:15 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 16 Acked 1015 Write 10 Done Done Done false 17 Acked 1016 Read 3 Done Done Done false 18 Acked 1017 Read 5 Done Done Done false 19 Acked 1018 Read 5 Done Done Done false 20 Acked 1019 Write 6 Done Done Done false 21 Acked 1020 Read 1 Done Done Done false 22 Acked 1021 Read 9 Done Done Done false 23 Acked 1022 Write 6 Done Done Done false 24 Acked 1023 Write 4 Done Done Done false 25 Acked 1024 Read 5 Done Done Done false 26 Acked 1025 Read 4 Done Done Done false 27 Acked 1026 Write 7 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 14 14 14 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1014 1014 1014 Downstairs last five completed: 1014 1013 1012 1011 1010 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:56:21.939018965Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1849} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 164 164 164 326 326 326 F F F 2 040-059 166 166 166 329 329 329 F F F 3 060-079 166 166 166 329 329 329 F F F 4 080-099 159 159 159 317 317 317 F F F 5 100-119 164 164 164 323 323 323 F F F 6 120-139 166 166 166 331 331 331 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 159 159 159 317 317 317 F F F 9 180-199 165 165 165 328 328 328 F F F 10 200-219 162 162 162 321 321 321 T T T 11 220-239 161 161 161 320 320 320 T T T 12 240-259 159 159 159 317 317 317 T T T 13 260-279 163 163 163 322 322 322 F F F 14 280-299 164 164 164 323 323 323 F F F 15 300-319 163 163 163 322 322 322 T T T 16 320-339 155 155 155 312 312 312 T T T 17 340-359 164 164 164 325 325 325 T T T 18 360-379 165 165 165 328 328 328 T T T 19 380-399 163 163 163 322 322 322 T T T 20 400-419 164 164 164 327 327 327 T T T 21 420-439 160 160 160 319 319 319 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 165 165 165 328 328 328 F F F 26 520-539 166 166 166 329 329 329 F F F 27 540-559 162 162 162 321 321 321 F F F 28 560-579 160 160 160 318 318 318 T T T 29 580-599 156 156 156 313 313 313 F F F Max gen: 166, Max flush: 331 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 164 164 164 326 326 326 F F F 2 040-059 166 166 166 329 329 329 F F F 3 060-079 166 166 166 329 329 329 F F F 4 080-099 159 159 159 317 317 317 F F F 5 100-119 164 164 164 323 323 323 F F F 6 120-139 166 166 166 331 331 331 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 159 159 159 317 317 317 F F F 9 180-199 165 165 165 328 328 328 F F F 10 200-219 167 167 167 332 332 332 F F F 11 220-239 167 167 167 332 332 332 F F F 12 240-259 167 167 167 332 332 332 F F F 13 260-279 163 163 163 322 322 322 F F F 14 280-299 164 164 164 323 323 323 F F F 15 300-319 167 167 167 332 332 332 F F F 16 320-339 167 167 167 332 332 332 F F F 17 340-359 167 167 167 332 332 332 F F F 18 360-379 167 167 167 332 332 332 F F F 19 380-399 167 167 167 332 332 332 F F F 20 400-419 167 167 167 332 332 332 F F F 21 420-439 160 160 160 319 319 319 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 165 165 165 328 328 328 F F F 26 520-539 166 166 166 329 329 329 F F F 27 540-559 162 162 162 321 321 321 F F F 28 560-579 167 167 167 332 332 332 F F F 29 580-599 156 156 156 313 313 313 F F F Max gen: 167, Max flush: 332 Begin loop 84 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.237012675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.237316814Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.237336452Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857} {"msg":"Crucible 4663ef81-80a3-4c23-8980-30c41bff3953 has session id: f4370fd3-509a-475e-8035-c05dceefc3f3","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.237782138Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Upstairs opts: Upstairs UUID: 4663ef81-80a3-4c23-8980-30c41bff3953, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.237829104Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Crucible stats registered with UUID: 4663ef81-80a3-4c23-8980-30c41bff3953","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.237856811Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238194997Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857} {"msg":"4663ef81-80a3-4c23-8980-30c41bff3953 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238225974Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238241463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238256191Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.23827021Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"1","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238284878Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"1","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238308416Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"2","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238339583Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"2","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238365821Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"io task","client":"2","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238391718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"io task","client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238410156Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"io task","client":"1","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:56:36.238759911Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"io task","client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:56:36.238799707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"io task","client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:56:36.238826985Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238841023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238854642Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238867991Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"io task","client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:49995)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.23897601Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"io task","client":"2","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:52605)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.238996348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"io task","client":"1","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.239696029Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"1","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.239719246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"1","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.239733035Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"2","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.239750473Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"2","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.240365352Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"1","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.24039406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"2","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.24089363Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"2","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.240920857Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"2","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.240938015Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.240959693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"1","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.240976672Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"1","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:36.24099326Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.24063707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"io task","client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:39457)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.240975217Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"io task","client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.242001565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.242026503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.242571029Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243169369Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243199926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[0]R flush_numbers[0..12]: [305, 326, 329, 329, 317, 323, 331, 331, 317, 328, 332, 332]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243218395Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[0]R generation[0..12]: [151, 164, 166, 166, 159, 164, 166, 166, 159, 165, 167, 167]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243236353Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243254221Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[1]R flush_numbers[0..12]: [305, 326, 329, 329, 317, 323, 331, 331, 317, 328, 332, 332]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243271669Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[1]R generation[0..12]: [151, 164, 166, 166, 159, 164, 166, 166, 159, 165, 167, 167]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243288828Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243306406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[2]R flush_numbers[0..12]: [305, 326, 329, 329, 317, 323, 331, 331, 317, 328, 332, 332]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243323634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[2]R generation[0..12]: [151, 164, 166, 166, 159, 164, 166, 166, 159, 165, 167, 167]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243340822Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2024-07-09T21:56:46.24336189Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Max found gen is 168","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243378499Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243394967Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Next flush: 333","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243411005Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243427214Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"":"downstairs","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243446162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.2434615Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"0","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243478169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"1","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243494477Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"client":"2","":"downstairs","session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243511306Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243526694Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"4663ef81-80a3-4c23-8980-30c41bff3953 is now active with session: f4370fd3-509a-475e-8035-c05dceefc3f3","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243542682Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243557781Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857,"session_id":"f4370fd3-509a-475e-8035-c05dceefc3f3"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:56:46.243573399Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 218 len 12288 data: 17 17 14 02/30 Write block 338 len 36864 data: 8 9 9 10 11 13 14 14 15 03/30 Read block 55 len 32768 04/30 Read block 205 len 36864 05/30 Write block 291 len 8192 data: 13 12 06/30 Flush 07/30 Write block 590 len 28672 data: 17 18 16 18 17 14 13 08/30 Read block 566 len 8192 09/30 Read block 140 len 4096 10/30 Write block 417 len 20480 data: 10 10 11 14 13 11/30 Read block 129 len 12288 12/30 Read block 68 len 4096 13/30 Read block 344 len 16384 14/30 Read block 366 len 32768 15/30 Read block 318 len 16384 16/30 Read block 340 len 4096 17/30 Write block 363 len 8192 data: 12 11 18/30 Read block 504 len 36864 19/30 Write block 45 len 36864 data: 14 12 13 12 11 7 10 10 16 20/30 Write block 359 len 12288 data: 18 17 14 21/30 Read block 593 len 4096 22/30 Flush 23/30 Write block 581 len 28672 data: 11 12 10 12 11 12 16 24/30 Read block 458 len 40960 25/30 Read block 528 len 4096 26/30 Read block 36 len 24576 27/30 Read block 564 len 32768 28/30 Read block 10 len 8192 29/30 Write block 98 len 4096 data: 11 30/30 Read block 62 len 16384 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 7 Done Done Done false 25 Acked 1024 Read 10 Done Done Done false 26 Acked 1025 Read 1 Done Done Done false 27 Acked 1026 Read 6 Done Done Done false 28 Acked 1027 Read 8 Done Done Done false 29 Acked 1028 Read 2 Done Done Done false 30 Acked 1029 Write 1 Done Done Sent false 31 Acked 1030 Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 2 3 Done 7 8 6 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:56:46.271489302Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1857} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 164 164 164 326 326 326 F F F 2 040-059 168 168 168 334 334 334 F F F 3 060-079 166 166 166 329 329 329 F F F 4 080-099 159 159 159 317 317 317 T T T 5 100-119 164 164 164 323 323 323 F F F 6 120-139 166 166 166 331 331 331 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 159 159 159 317 317 317 F F F 9 180-199 165 165 165 328 328 328 F F F 10 200-219 168 168 168 333 333 333 F F F 11 220-239 168 168 168 333 333 333 F F F 12 240-259 167 167 167 332 332 332 F F F 13 260-279 163 163 163 322 322 322 F F F 14 280-299 168 168 168 333 333 333 F F F 15 300-319 167 167 167 332 332 332 F F F 16 320-339 168 168 168 333 333 333 F F F 17 340-359 168 168 168 335 335 335 F F F 18 360-379 168 168 168 335 335 335 F F F 19 380-399 167 167 167 332 332 332 F F F 20 400-419 168 168 168 334 334 334 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 165 165 165 328 328 328 F F F 26 520-539 166 166 166 329 329 329 F F F 27 540-559 162 162 162 321 321 321 F F F 28 560-579 167 167 167 332 332 332 F F F 29 580-599 168 168 168 334 334 334 T T T Max gen: 168, Max flush: 335 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 164 164 164 326 326 326 F F F 2 040-059 168 168 168 334 334 334 F F F 3 060-079 166 166 166 329 329 329 F F F 4 080-099 169 169 169 336 336 336 F F F 5 100-119 164 164 164 323 323 323 F F F 6 120-139 166 166 166 331 331 331 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 159 159 159 317 317 317 F F F 9 180-199 165 165 165 328 328 328 F F F 10 200-219 168 168 168 333 333 333 F F F 11 220-239 168 168 168 333 333 333 F F F 12 240-259 167 167 167 332 332 332 F F F 13 260-279 163 163 163 322 322 322 F F F 14 280-299 168 168 168 333 333 333 F F F 15 300-319 167 167 167 332 332 332 F F F 16 320-339 168 168 168 333 333 333 F F F 17 340-359 168 168 168 335 335 335 F F F 18 360-379 168 168 168 335 335 335 F F F 19 380-399 167 167 167 332 332 332 F F F 20 400-419 168 168 168 334 334 334 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 153 153 153 308 308 308 F F F 25 500-519 165 165 165 328 328 328 F F F 26 520-539 166 166 166 329 329 329 F F F 27 540-559 162 162 162 321 321 321 F F F 28 560-579 167 167 167 332 332 332 F F F 29 580-599 169 169 169 336 336 336 F F F Max gen: 169, Max flush: 336 Begin loop 85 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.455337734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.455913477Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.455941134Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865} {"msg":"Crucible 5feabadc-0268-4bf2-a3cd-0377a56579d0 has session id: 7eebebdc-d556-4549-a4d2-a3729c8bf082","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456099479Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Upstairs opts: Upstairs UUID: 5feabadc-0268-4bf2-a3cd-0377a56579d0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456142625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Crucible stats registered with UUID: 5feabadc-0268-4bf2-a3cd-0377a56579d0","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456167162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456528866Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865} {"msg":"5feabadc-0268-4bf2-a3cd-0377a56579d0 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456557513Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456581511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456608288Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456631566Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"1","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456654934Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"1","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456678161Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"2","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456706309Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"2","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456731626Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"io task","client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456754754Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"io task","client":"1","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456777842Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"io task","client":"2","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:00.456840996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"io task","client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"ds_connection connected from Ok(127.0.0.1:41003)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456874922Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"io task","client":"2","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"ds_connection connected from Ok(127.0.0.1:47511)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.456901959Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"io task","client":"1","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:00.456931236Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"io task","client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:00.456960174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.457045945Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.457071103Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.457172303Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"io task","client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.457680812Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"2","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.45770508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"2","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.457721838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"1","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.457739116Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"1","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.458341597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"2","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.458378703Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"1","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.458938457Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"2","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.458975784Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"2","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.459004531Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.459022839Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"1","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.459037958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"1","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:00.459051996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.458864241Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"io task","client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"ds_connection connected from Ok(127.0.0.1:50774)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.458962371Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"io task","client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.460140684Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.460168151Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.460733846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461315408Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461347745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[0]R flush_numbers[0..12]: [305, 326, 334, 329, 336, 323, 331, 331, 317, 328, 333, 333]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461366703Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[0]R generation[0..12]: [151, 164, 168, 166, 169, 164, 166, 166, 159, 165, 168, 168]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461384661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461402639Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[1]R flush_numbers[0..12]: [305, 326, 334, 329, 336, 323, 331, 331, 317, 328, 333, 333]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461420398Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[1]R generation[0..12]: [151, 164, 168, 166, 169, 164, 166, 166, 159, 165, 168, 168]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461438226Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461456194Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[2]R flush_numbers[0..12]: [305, 326, 334, 329, 336, 323, 331, 331, 317, 328, 333, 333]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461473622Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[2]R generation[0..12]: [151, 164, 168, 166, 169, 164, 166, 166, 159, 165, 168, 168]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:57:10.461491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461509189Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461525577Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461542145Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Next flush: 337","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461558294Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461574652Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"":"downstairs","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.46159348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461609079Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"0","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461625627Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"1","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461642276Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"client":"2","":"downstairs","session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461659044Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461674372Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"5feabadc-0268-4bf2-a3cd-0377a56579d0 is now active with session: 7eebebdc-d556-4549-a4d2-a3729c8bf082","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461690671Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461705599Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865,"session_id":"7eebebdc-d556-4549-a4d2-a3729c8bf082"} {"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:10.461720908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 266 len 12288 data: 13 12 9 02/30 Write block 527 len 40960 data: 13 13 12 9 12 10 10 9 10 11 03/30 Read block 158 len 36864 04/30 Read block 69 len 16384 05/30 Read block 263 len 8192 06/30 Write block 485 len 4096 data: 15 07/30 Read block 520 len 24576 08/30 Read block 166 len 12288 09/30 Read block 120 len 40960 10/30 Write block 524 len 40960 data: 10 9 13 14 14 13 10 13 11 11 11/30 Write block 369 len 24576 data: 18 15 12 14 16 16 12/30 Write block 489 len 36864 data: 16 15 14 7 7 7 6 4 5 13/30 Read block 537 len 12288 14/30 Write block 172 len 20480 data: 6 5 5 12 10 15/30 Read block 316 len 4096 16/30 Write block 125 len 40960 data: 17 16 10 8 8 7 9 9 11 12 17/30 Read block 279 len 40960 18/30 Write block 114 len 20480 data: 19 17 14 13 8 19/30 Read block 350 len 16384 20/30 Write block 527 len 20480 data: 15 15 14 11 14 21/30 Write block 280 len 40960 data: 12 11 10 13 14 14 12 9 12 14 22/30 Flush 23/30 Write block 502 len 28672 data: 7 8 9 10 13 12 14 24/30 Read block 549 len 8192 25/30 Read block 386 len 12288 26/30 Read block 66 len 24576 27/30 Read block 540 len 4096 28/30 Write block 321 len 40960 data: 17 15 14 12 12 11 11 10 10 11 29/30 Read block 182 len 36864 30/30 Write block 22 len 40960 data: 11 10 11 11 11 12 14 16 15 14 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 7 Done Done Done false 25 Acked 1024 Read 2 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Read 6 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Write 10 Done Done Done false 30 Acked 1029 Read 9 Done Done Done false 31 Acked 1030 Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:10.490346091Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1865} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 164 164 164 326 326 326 T T T 2 040-059 168 168 168 334 334 334 F F F 3 060-079 166 166 166 329 329 329 F F F 4 080-099 169 169 169 336 336 336 F F F 5 100-119 170 170 170 337 337 337 F F F 6 120-139 170 170 170 337 337 337 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 170 170 170 337 337 337 F F F 9 180-199 165 165 165 328 328 328 F F F 10 200-219 168 168 168 333 333 333 F F F 11 220-239 168 168 168 333 333 333 F F F 12 240-259 167 167 167 332 332 332 F F F 13 260-279 170 170 170 337 337 337 F F F 14 280-299 170 170 170 338 338 338 F F F 15 300-319 167 167 167 332 332 332 F F F 16 320-339 168 168 168 333 333 333 T T T 17 340-359 168 168 168 335 335 335 F F F 18 360-379 170 170 170 337 337 337 F F F 19 380-399 167 167 167 332 332 332 F F F 20 400-419 168 168 168 334 334 334 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 170 170 170 337 337 337 F F F 25 500-519 165 165 165 328 328 328 T T T 26 520-539 170 170 170 337 337 337 F F F 27 540-559 162 162 162 321 321 321 F F F 28 560-579 167 167 167 332 332 332 F F F 29 580-599 169 169 169 336 336 336 F F F Max gen: 170, Max flush: 338 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 171 171 171 339 339 339 F F F 2 040-059 168 168 168 334 334 334 F F F 3 060-079 166 166 166 329 329 329 F F F 4 080-099 169 169 169 336 336 336 F F F 5 100-119 170 170 170 337 337 337 F F F 6 120-139 170 170 170 337 337 337 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 170 170 170 337 337 337 F F F 9 180-199 165 165 165 328 328 328 F F F 10 200-219 168 168 168 333 333 333 F F F 11 220-239 168 168 168 333 333 333 F F F 12 240-259 167 167 167 332 332 332 F F F 13 260-279 170 170 170 337 337 337 F F F 14 280-299 170 170 170 338 338 338 F F F 15 300-319 167 167 167 332 332 332 F F F 16 320-339 171 171 171 339 339 339 F F F 17 340-359 168 168 168 335 335 335 F F F 18 360-379 170 170 170 337 337 337 F F F 19 380-399 167 167 167 332 332 332 F F F 20 400-419 168 168 168 334 334 334 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 157 157 157 314 314 314 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 170 170 170 337 337 337 F F F 25 500-519 171 171 171 339 339 339 F F F 26 520-539 170 170 170 337 337 337 F F F 27 540-559 162 162 162 321 321 321 F F F 28 560-579 167 167 167 332 332 332 F F F 29 580-599 169 169 169 336 336 336 F F F Max gen: 171, Max flush: 339 Begin loop 86 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.69186236Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.692138093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.692156661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873} {"msg":"Crucible 50ee04d5-d73a-43f2-85df-1b1a51c06148 has session id: 4cecf40c-a0ba-4ae2-aded-b170448a119f","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.692688129Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Upstairs opts: Upstairs UUID: 50ee04d5-d73a-43f2-85df-1b1a51c06148, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.692733404Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Crucible stats registered with UUID: 50ee04d5-d73a-43f2-85df-1b1a51c06148","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.692752992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693070421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873} {"msg":"50ee04d5-d73a-43f2-85df-1b1a51c06148 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693156462Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693174901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"0","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693189469Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"0","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693203118Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"1","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693216796Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"1","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693236914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693261542Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693284899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"io task","client":"0","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693309697Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"io task","client":"1","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693325775Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"io task","client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"ds_connection connected from Ok(127.0.0.1:34522)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693507757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"io task","client":"0","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:24.693544724Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"io task","client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:24.693607698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"io task","client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:24.693643214Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693668052Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693694889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"ds_connection connected from Ok(127.0.0.1:61400)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693721736Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"io task","client":"1","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.693743384Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"io task","client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.694329576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"0","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.694367332Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"0","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.69439078Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"1","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.694412668Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"1","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.694977572Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"1","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.695010119Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"0","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.695491191Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"1","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.695514829Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"1","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.695528837Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.69560399Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"0","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.695622688Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"0","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:24.695636286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.69482944Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"io task","client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"ds_connection connected from Ok(127.0.0.1:50493)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.695047359Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"io task","client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.696379437Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.696405874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697036032Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697620564Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697641841Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[0]R flush_numbers[0..12]: [305, 339, 334, 329, 336, 337, 337, 331, 337, 328, 333, 333]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.69765749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[0]R generation[0..12]: [151, 171, 168, 166, 169, 170, 170, 166, 170, 165, 168, 168]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697672469Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697687027Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[1]R flush_numbers[0..12]: [305, 339, 334, 329, 336, 337, 337, 331, 337, 328, 333, 333]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697701156Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[1]R generation[0..12]: [151, 171, 168, 166, 169, 170, 170, 166, 170, 165, 168, 168]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697715224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697729423Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[2]R flush_numbers[0..12]: [305, 339, 334, 329, 336, 337, 337, 331, 337, 328, 333, 333]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697743442Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[2]R generation[0..12]: [151, 171, 168, 166, 169, 170, 170, 166, 170, 165, 168, 168]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.69775716Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697771169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2024-07-09T21:57:34.697784098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Generation requested: 172 >= found:172","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697801606Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Next flush: 340","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697818784Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697832203Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"":"downstairs","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697845261Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.69785734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"0","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697870899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"1","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697884657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"client":"2","":"downstairs","session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697898336Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697910715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"50ee04d5-d73a-43f2-85df-1b1a51c06148 is now active with session: 4cecf40c-a0ba-4ae2-aded-b170448a119f","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697923674Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697935472Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873,"session_id":"4cecf40c-a0ba-4ae2-aded-b170448a119f"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:34.697947821Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 237 len 40960 data: 11 12 8 12 13 13 12 10 8 6 02/30 Flush 03/30 Write block 546 len 28672 data: 15 16 16 16 17 15 17 04/30 Write block 26 len 24576 data: 12 13 15 17 16 15 05/30 Read block 210 len 16384 06/30 Write block 166 len 12288 data: 13 11 8 07/30 Read block 158 len 8192 08/30 Read block 390 len 24576 09/30 Flush 10/30 Write block 524 len 12288 data: 11 10 14 11/30 Read block 429 len 24576 12/30 Write block 174 len 28672 data: 6 13 11 13 11 9 8 13/30 Write block 295 len 20480 data: 11 10 11 12 12 14/30 Read block 217 len 28672 15/30 Flush 16/30 Write block 518 len 24576 data: 10 9 11 8 6 8 17/30 Read block 529 len 16384 18/30 Read block 102 len 40960 19/30 Read block 75 len 20480 20/30 Read block 394 len 24576 21/30 Read block 333 len 36864 22/30 Write block 440 len 36864 data: 13 15 15 15 15 15 12 13 15 23/30 Write block 223 len 12288 data: 16 14 13 24/30 Read block 117 len 16384 25/30 Write block 271 len 20480 data: 10 9 9 8 10 26/30 Flush 27/30 Write block 205 len 12288 data: 20 21 20 28/30 Read block 598 len 4096 29/30 Read block 108 len 28672 30/30 Read block 578 len 36864 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 3 Done Done Done false 29 Acked 1028 Read 1 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 4 3 3 10 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:34.731124003Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1873} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 172 172 172 341 341 341 F F F 2 040-059 168 168 168 334 334 334 F F F 3 060-079 166 166 166 329 329 329 F F F 4 080-099 169 169 169 336 336 336 F F F 5 100-119 170 170 170 337 337 337 F F F 6 120-139 170 170 170 337 337 337 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 172 172 172 342 342 342 F F F 9 180-199 172 172 172 342 342 342 F F F 10 200-219 168 168 168 333 333 333 T T T 11 220-239 172 172 172 344 344 344 F F F 12 240-259 172 172 172 340 340 340 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 172 172 172 343 343 343 F F F 15 300-319 167 167 167 332 332 332 F F F 16 320-339 171 171 171 339 339 339 F F F 17 340-359 168 168 168 335 335 335 F F F 18 360-379 170 170 170 337 337 337 F F F 19 380-399 167 167 167 332 332 332 F F F 20 400-419 168 168 168 334 334 334 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 172 172 172 344 344 344 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 170 170 170 337 337 337 F F F 25 500-519 172 172 172 344 344 344 F F F 26 520-539 172 172 172 344 344 344 F F F 27 540-559 172 172 172 341 341 341 F F F 28 560-579 167 167 167 332 332 332 F F F 29 580-599 169 169 169 336 336 336 F F F Max gen: 172, Max flush: 344 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 172 172 172 341 341 341 F F F 2 040-059 168 168 168 334 334 334 F F F 3 060-079 166 166 166 329 329 329 F F F 4 080-099 169 169 169 336 336 336 F F F 5 100-119 170 170 170 337 337 337 F F F 6 120-139 170 170 170 337 337 337 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 172 172 172 342 342 342 F F F 9 180-199 172 172 172 342 342 342 F F F 10 200-219 173 173 173 345 345 345 F F F 11 220-239 172 172 172 344 344 344 F F F 12 240-259 172 172 172 340 340 340 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 172 172 172 343 343 343 F F F 15 300-319 167 167 167 332 332 332 F F F 16 320-339 171 171 171 339 339 339 F F F 17 340-359 168 168 168 335 335 335 F F F 18 360-379 170 170 170 337 337 337 F F F 19 380-399 167 167 167 332 332 332 F F F 20 400-419 168 168 168 334 334 334 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 172 172 172 344 344 344 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 170 170 170 337 337 337 F F F 25 500-519 172 172 172 344 344 344 F F F 26 520-539 172 172 172 344 344 344 F F F 27 540-559 172 172 172 341 341 341 F F F 28 560-579 167 167 167 332 332 332 F F F 29 580-599 169 169 169 336 336 336 F F F Max gen: 173, Max flush: 345 Begin loop 87 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.901085772Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.901370914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.901387922Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881} {"msg":"Crucible 92e3661b-8f54-4763-b52b-81463e199d02 has session id: d662a93b-4b53-4898-9f1f-01767f6dd137","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.901890492Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Upstairs opts: Upstairs UUID: 92e3661b-8f54-4763-b52b-81463e199d02, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.901934988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Crucible stats registered with UUID: 92e3661b-8f54-4763-b52b-81463e199d02","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.901958616Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902344898Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881} {"msg":"92e3661b-8f54-4763-b52b-81463e199d02 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902376204Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902392003Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"0","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902407301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"0","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.90242313Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902447017Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902476725Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"2","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.90251631Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"2","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902543408Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"io task","client":"2","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902568155Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"io task","client":"0","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902591903Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"io task","client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:44.902704882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"io task","client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"ds_connection connected from Ok(127.0.0.1:58284)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902741418Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"io task","client":"2","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:44.902771195Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"io task","client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"ds_connection connected from Ok(127.0.0.1:46093)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902797383Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"io task","client":"0","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:44.902878874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902902792Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.90292605Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.902966126Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"io task","client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.903510972Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"0","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.903545928Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"0","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.903577685Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"2","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.903604933Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"2","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.904127071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"2","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.904155068Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"0","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.904620922Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"0","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.904646419Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"0","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.904661008Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.904681696Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"2","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.904695694Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"2","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:44.904709143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.9041004Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"io task","client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"ds_connection connected from Ok(127.0.0.1:61560)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.904349245Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"io task","client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.905942717Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.905984993Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.906509421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907032509Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907058766Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[0]R flush_numbers[0..12]: [305, 341, 334, 329, 336, 337, 337, 331, 342, 342, 345, 344]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907075135Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[0]R generation[0..12]: [151, 172, 168, 166, 169, 170, 170, 166, 172, 172, 173, 172]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907089913Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907104482Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[1]R flush_numbers[0..12]: [305, 341, 334, 329, 336, 337, 337, 331, 342, 342, 345, 344]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907119Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[1]R generation[0..12]: [151, 172, 168, 166, 169, 170, 170, 166, 172, 172, 173, 172]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907133419Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907147678Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[2]R flush_numbers[0..12]: [305, 341, 334, 329, 336, 337, 337, 331, 342, 342, 345, 344]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907161846Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[2]R generation[0..12]: [151, 172, 168, 166, 169, 170, 170, 166, 172, 172, 173, 172]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907187023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":Wait for a query_work_queue command to finish before sending IO 30,"time":"2024-07-09T21:57:54.90722081Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907251187Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907280464Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Next flush: 346","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907305122Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907337319Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"":"downstairs","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907368765Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907397233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"0","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907416001Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"1","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907442728Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"client":"2","":"downstairs","session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907474005Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907502872Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"92e3661b-8f54-4763-b52b-81463e199d02 is now active with session: d662a93b-4b53-4898-9f1f-01767f6dd137","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.90752101Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907544738Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881,"session_id":"d662a93b-4b53-4898-9f1f-01767f6dd137"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:57:54.907572755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 284 len 40960 data: 15 15 13 10 13 15 13 14 13 10 02/30 Read block 137 len 20480 03/30 Write block 499 len 24576 data: 5 7 6 8 9 10 04/30 Write block 198 len 40960 data: 14 14 14 10 12 15 18 21 22 21 05/30 Read block 425 len 36864 06/30 Write block 85 len 12288 data: 16 17 16 07/30 Read block 453 len 8192 08/30 Write block 305 len 24576 data: 15 15 12 12 13 13 09/30 Write block 230 len 16384 data: 9 10 9 10 10/30 Write block 563 len 32768 data: 12 12 10 9 8 14 13 16 11/30 Read block 362 len 16384 12/30 Flush 13/30 Flush 14/30 Write block 46 len 4096 data: 13 15/30 Read block 63 len 16384 16/30 Read block 523 len 4096 17/30 Flush 18/30 Flush 19/30 Write block 74 len 36864 data: 13 10 11 12 13 10 12 13 15 20/30 Write block 581 len 16384 data: 12 13 11 13 21/30 Write block 412 len 32768 data: 13 11 11 12 10 11 11 12 22/30 Write block 353 len 28672 data: 11 8 9 13 16 20 19 23/30 Write block 515 len 24576 data: 11 9 12 11 10 12 24/30 Read block 573 len 8192 25/30 Read block 244 len 4096 26/30 Read block 193 len 4096 27/30 Write block 137 len 4096 data: 12 28/30 Write block 223 len 8192 data: 17 15 29/30 Write block 480 len 32768 data: 9 9 11 11 13 16 17 20 30/30 Write block 45 len 12288 data: 15 14 14 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 9 Done Done Done false 20 Acked 1019 Write 4 Done Done Done false 21 Acked 1020 Write 8 Done Done Done false 22 Acked 1021 Write 7 Done Done Done false 23 Acked 1022 Write 6 Done Done Done false 24 Acked 1023 Read 2 Done Done Done false 25 Acked 1024 Read 1 Done Done Done false 26 Acked 1025 Read 1 Done Done Done false 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Write 8 Done Done Sent false 30 Acked 1029 Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 2 4 Done 11 11 10 32 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:57:54.934341412Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1881} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 172 172 172 341 341 341 F F F 2 040-059 174 174 174 348 348 348 T T T 3 060-079 166 166 166 329 329 329 T T T 4 080-099 174 174 174 346 346 346 T T T 5 100-119 170 170 170 337 337 337 F F F 6 120-139 170 170 170 337 337 337 T T T 7 140-159 166 166 166 331 331 331 F F F 8 160-179 172 172 172 342 342 342 F F F 9 180-199 174 174 174 346 346 346 F F F 10 200-219 174 174 174 346 346 346 F F F 11 220-239 174 174 174 346 346 346 T T T 12 240-259 172 172 172 340 340 340 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 174 174 174 346 346 346 F F F 15 300-319 174 174 174 346 346 346 F F F 16 320-339 171 171 171 339 339 339 F F F 17 340-359 168 168 168 335 335 335 T T T 18 360-379 170 170 170 337 337 337 F F F 19 380-399 167 167 167 332 332 332 F F F 20 400-419 168 168 168 334 334 334 T T T 21 420-439 168 168 168 334 334 334 F F F 22 440-459 172 172 172 344 344 344 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 174 174 174 346 346 346 T T T 25 500-519 174 174 174 346 346 346 T T T 26 520-539 172 172 172 344 344 344 T T T 27 540-559 172 172 172 341 341 341 F F F 28 560-579 174 174 174 346 346 346 F F F 29 580-599 169 169 169 336 336 336 T T T Max gen: 174, Max flush: 348 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 172 172 172 341 341 341 F F F 2 040-059 175 175 175 349 349 349 F F F 3 060-079 175 175 175 349 349 349 F F F 4 080-099 175 175 175 349 349 349 F F F 5 100-119 170 170 170 337 337 337 F F F 6 120-139 175 175 175 349 349 349 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 172 172 172 342 342 342 F F F 9 180-199 174 174 174 346 346 346 F F F 10 200-219 174 174 174 346 346 346 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 172 172 172 340 340 340 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 174 174 174 346 346 346 F F F 15 300-319 174 174 174 346 346 346 F F F 16 320-339 171 171 171 339 339 339 F F F 17 340-359 175 175 175 349 349 349 F F F 18 360-379 170 170 170 337 337 337 F F F 19 380-399 167 167 167 332 332 332 F F F 20 400-419 175 175 175 349 349 349 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 172 172 172 344 344 344 F F F 23 460-479 158 158 158 315 315 315 F F F 24 480-499 175 175 175 349 349 349 F F F 25 500-519 175 175 175 349 349 349 F F F 26 520-539 175 175 175 349 349 349 F F F 27 540-559 172 172 172 341 341 341 F F F 28 560-579 174 174 174 346 346 346 F F F 29 580-599 175 175 175 349 349 349 F F F Max gen: 175, Max flush: 349 Begin loop 88 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.248069603Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.248560814Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.248589072Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889} {"msg":"Crucible 4a5bcbac-08d8-4a5a-8b45-6d88cd456dff has session id: 8501e05a-a1ef-499a-8c8c-272a44a92945","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.248780473Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Upstairs opts: Upstairs UUID: 4a5bcbac-08d8-4a5a-8b45-6d88cd456dff, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.248824378Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Crucible stats registered with UUID: 4a5bcbac-08d8-4a5a-8b45-6d88cd456dff","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.248849066Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249178033Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889} {"msg":"4a5bcbac-08d8-4a5a-8b45-6d88cd456dff active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249202471Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249217129Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"0","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249232148Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"0","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249259415Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249285372Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.24930882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"2","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249331508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"2","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249359485Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"io task","client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249386282Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"io task","client":"0","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.24941055Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"io task","client":"2","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:09.249770315Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"io task","client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"ds_connection connected from Ok(127.0.0.1:38980)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249805341Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"io task","client":"0","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:09.249830598Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"io task","client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:09.249856596Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249877084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249952137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"ds_connection connected from Ok(127.0.0.1:34899)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.249988413Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"io task","client":"2","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.250042937Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"io task","client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.250284453Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"0","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.25032154Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"0","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.250345938Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"2","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.250367965Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"2","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.25092348Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"2","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.250951378Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"0","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.251439389Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"2","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.251476335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"2","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.251492374Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.251505043Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"0","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.251518791Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"0","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:09.25153225Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.251338615Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"io task","client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"ds_connection connected from Ok(127.0.0.1:36949)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.251628616Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"io task","client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.25260218Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.252632617Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253165054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253679763Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253697851Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[0]R flush_numbers[0..12]: [305, 341, 349, 349, 349, 337, 349, 331, 342, 346, 346, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253719769Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[0]R generation[0..12]: [151, 172, 175, 175, 175, 170, 175, 166, 172, 174, 174, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253734427Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253748706Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[1]R flush_numbers[0..12]: [305, 341, 349, 349, 349, 337, 349, 331, 342, 346, 346, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253762745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[1]R generation[0..12]: [151, 172, 175, 175, 175, 170, 175, 166, 172, 174, 174, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253776673Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253790852Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[2]R flush_numbers[0..12]: [305, 341, 349, 349, 349, 337, 349, 331, 342, 346, 346, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.25380476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[2]R generation[0..12]: [151, 172, 175, 175, 175, 170, 175, 166, 172, 174, 174, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253818729Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253832798Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Max found gen is 176","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253845876Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Wait for a query_work_queue command to finish before sending IO Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253864885Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Next flush: 350","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253879103Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253892362Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"":"downstairs","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253905541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253917649Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"0","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253931228Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"1","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253944257Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"client":"2","":"downstairs","session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253957695Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253970034Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"4a5bcbac-08d8-4a5a-8b45-6d88cd456dff is now active with session: 8501e05a-a1ef-499a-8c8c-272a44a92945","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253982913Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.253994882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889,"session_id":"8501e05a-a1ef-499a-8c8c-272a44a92945"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:19.25400718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 499 len 32768 data: 6 8 7 9 10 11 11 14 02/30 Read block 96 len 28672 03/30 Read block 504 len 36864 04/30 Read block 491 len 16384 05/30 Read block 258 len 8192 06/30 Write block 118 len 24576 data: 9 10 9 9 11 15 07/30 Write block 517 len 28672 data: 13 12 11 13 9 7 9 08/30 Write block 182 len 28672 data: 7 9 7 8 8 11 12 09/30 Write block 338 len 24576 data: 9 10 10 11 12 14 10/30 Write block 255 len 20480 data: 10 11 10 9 10 11/30 Read block 167 len 36864 12/30 Read block 46 len 36864 13/30 Write block 298 len 12288 data: 13 13 15 14/30 Read block 51 len 20480 15/30 Read block 105 len 32768 16/30 Write block 398 len 12288 data: 9 12 13 17/30 Flush 18/30 Write block 470 len 24576 data: 11 8 9 9 7 5 19/30 Read block 422 len 4096 20/30 Read block 349 len 16384 21/30 Read block 314 len 12288 22/30 Write block 59 len 28672 data: 14 13 15 17 14 12 14 23/30 Write block 513 len 4096 data: 12 24/30 Write block 399 len 8192 data: 13 14 25/30 Read block 269 len 24576 26/30 Write block 319 len 12288 data: 17 16 18 27/30 Write block 492 len 20480 data: 8 8 8 7 5 28/30 Write block 450 len 40960 data: 16 17 17 16 17 16 13 13 12 10 29/30 Read block 466 len 4096 30/30 Write block 574 len 4096 data: 17 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 6 Done Done Done false 19 Acked 1018 Read 1 Done Done Done false 20 Acked 1019 Read 4 Done Done Done false 21 Acked 1020 Read 3 Done Done Done false 22 Acked 1021 Write 7 Done Done Done false 23 Acked 1022 Write 1 Done Done Done false 24 Acked 1023 Write 2 Done Done Done false 25 Acked 1024 Read 6 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Write 5 Done Done Done false 28 Acked 1027 Write 10 Done Done Done false 29 Acked 1028 Read 1 Done Done Done false 30 Acked 1029 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 12 12 12 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:19.279584505Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1889} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 172 172 172 341 341 341 F F F 2 040-059 175 175 175 349 349 349 T T T 3 060-079 175 175 175 349 349 349 T T T 4 080-099 175 175 175 349 349 349 F F F 5 100-119 176 176 176 350 350 350 F F F 6 120-139 176 176 176 350 350 350 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 172 172 172 342 342 342 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 174 174 174 346 346 346 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 176 176 176 350 350 350 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 176 176 176 350 350 350 F F F 15 300-319 176 176 176 350 350 350 T T T 16 320-339 176 176 176 350 350 350 T T T 17 340-359 176 176 176 350 350 350 F F F 18 360-379 170 170 170 337 337 337 F F F 19 380-399 176 176 176 350 350 350 T T T 20 400-419 176 176 176 350 350 350 T T T 21 420-439 168 168 168 334 334 334 F F F 22 440-459 172 172 172 344 344 344 T T T 23 460-479 158 158 158 315 315 315 T T T 24 480-499 176 176 176 350 350 350 T T T 25 500-519 176 176 176 350 350 350 T T T 26 520-539 176 176 176 350 350 350 F F F 27 540-559 172 172 172 341 341 341 F F F 28 560-579 174 174 174 346 346 346 T T T 29 580-599 175 175 175 349 349 349 F F F Max gen: 176, Max flush: 350 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 305 305 305 F F F 1 020-039 172 172 172 341 341 341 F F F 2 040-059 177 177 177 351 351 351 F F F 3 060-079 177 177 177 351 351 351 F F F 4 080-099 175 175 175 349 349 349 F F F 5 100-119 176 176 176 350 350 350 F F F 6 120-139 176 176 176 350 350 350 F F F 7 140-159 166 166 166 331 331 331 F F F 8 160-179 172 172 172 342 342 342 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 174 174 174 346 346 346 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 176 176 176 350 350 350 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 176 176 176 350 350 350 F F F 15 300-319 177 177 177 351 351 351 F F F 16 320-339 177 177 177 351 351 351 F F F 17 340-359 176 176 176 350 350 350 F F F 18 360-379 170 170 170 337 337 337 F F F 19 380-399 177 177 177 351 351 351 F F F 20 400-419 177 177 177 351 351 351 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 177 177 177 351 351 351 F F F 23 460-479 177 177 177 351 351 351 F F F 24 480-499 177 177 177 351 351 351 F F F 25 500-519 177 177 177 351 351 351 F F F 26 520-539 176 176 176 350 350 350 F F F 27 540-559 172 172 172 341 341 341 F F F 28 560-579 177 177 177 351 351 351 F F F 29 580-599 175 175 175 349 349 349 F F F Max gen: 177, Max flush: 351 Begin loop 89 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.589414932Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.589724841Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.589752629Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897} {"msg":"Crucible 839152e7-3ce3-417f-ad17-b630454b8240 has session id: 833af570-19d7-4031-82ae-64b76305b1c9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590074347Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Upstairs opts: Upstairs UUID: 839152e7-3ce3-417f-ad17-b630454b8240, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590118093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Crucible stats registered with UUID: 839152e7-3ce3-417f-ad17-b630454b8240","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.59014142Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.5904457Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897} {"msg":"839152e7-3ce3-417f-ad17-b630454b8240 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590513853Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590539601Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"0","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590566538Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"0","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590592295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"1","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590616203Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"1","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.59064147Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590667008Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590690506Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"io task","client":"1","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590713973Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"io task","client":"0","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590737331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"io task","client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"ds_connection connected from Ok(127.0.0.1:51522)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.590930892Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"io task","client":"0","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:33.590973218Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"io task","client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:33.591006274Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"io task","client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:33.591037661Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.591061699Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.591086506Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"ds_connection connected from Ok(127.0.0.1:48966)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.591110184Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"io task","client":"1","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.591168508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"io task","client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.591751291Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"0","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.591777948Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"0","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.591792816Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"1","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.591809325Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"1","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.592449691Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"1","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.59256617Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"0","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.592887018Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"1","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.592910756Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"1","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.592925884Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.593073789Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"0","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.593092908Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"0","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:33.593107776Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.591961915Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"io task","client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"ds_connection connected from Ok(127.0.0.1:60137)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.592269584Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"io task","client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.593413181Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.593441328Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.593971715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594510042Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594538159Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[0]R flush_numbers[0..12]: [305, 341, 351, 351, 349, 350, 350, 331, 342, 350, 346, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594556568Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[0]R generation[0..12]: [151, 172, 177, 177, 175, 176, 176, 166, 172, 176, 174, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594574816Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594593024Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[1]R flush_numbers[0..12]: [305, 341, 351, 351, 349, 350, 350, 331, 342, 350, 346, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594610722Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[1]R generation[0..12]: [151, 172, 177, 177, 175, 176, 176, 166, 172, 176, 174, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.59462819Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594645719Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[2]R flush_numbers[0..12]: [305, 341, 351, 351, 349, 350, 350, 331, 342, 350, 346, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594663227Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_idWait for a query_work_queue command to finish before sending IO ":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[2]R generation[0..12]: [151, 172, 177, 177, 175, 176, 176, 166, 172, 176, 174, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594684295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594706933Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594723471Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594740079Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Next flush: 352","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594756598Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594772926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"":"downstairs","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594789674Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594805043Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"0","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594818511Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"1","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.59483172Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"client":"2","":"downstairs","session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594845009Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594857328Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"839152e7-3ce3-417f-ad17-b630454b8240 is now active with session: 833af570-19d7-4031-82ae-64b76305b1c9","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594870366Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594882295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897,"session_id":"833af570-19d7-4031-82ae-64b76305b1c9"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:43.594894694Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 364 len 36864 data: 12 14 16 14 14 19 16 13 15 02/30 Read block 588 len 8192 03/30 Write block 127 len 36864 data: 11 9 9 8 10 10 12 13 10 04/30 Write block 359 len 8192 data: 20 18 05/30 Read block 66 len 36864 06/30 Write block 151 len 40960 data: 12 10 9 10 10 9 7 8 8 12 07/30 Read block 296 len 24576 08/30 Read block 453 len 36864 09/30 Flush 10/30 Write block 572 len 16384 data: 19 19 18 15 11/30 Flush 12/30 Write block 128 len 36864 data: 10 10 9 11 11 13 14 11 9 13/30 Write block 532 len 36864 data: 12 12 10 11 12 11 14 13 13 14/30 Read block 107 len 24576 15/30 Write block 251 len 20480 data: 8 8 8 7 11 16/30 Write block 338 len 20480 data: 10 11 11 12 13 17/30 Read block 127 len 20480 18/30 Write block 527 len 36864 data: 16 16 15 12 15 13 13 11 12 19/30 Read block 586 len 16384 20/30 Read block 462 len 32768 21/30 Write block 28 len 36864 data: 16 18 17 16 16 14 15 16 17 22/30 Read block 263 len 16384 23/30 Write block 12 len 36864 data: 16 16 17 15 15 15 11 9 10 24/30 Read block 308 len 4096 25/30 Write block 483 len 36864 data: 12 14 17 18 21 18 17 16 15 26/30 Flush 27/30 Write block 471 len 8192 data: 9 10 28/30 Read block 103 len 20480 29/30 Write block 372 len 24576 data: 16 17 17 16 15 10 30/30 Read block 481 len 24576 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Write 6 Done Done Done false 31 Acked 1030 Read 6 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:43.626871145Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1897} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 355 355 355 F F F 1 020-039 178 178 178 355 355 355 F F F 2 040-059 177 177 177 351 351 351 F F F 3 060-079 177 177 177 351 351 351 F F F 4 080-099 175 175 175 349 349 349 F F F 5 100-119 176 176 176 350 350 350 F F F 6 120-139 178 178 178 354 354 354 F F F 7 140-159 178 178 178 352 352 352 F F F 8 160-179 178 178 178 352 352 352 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 174 174 174 346 346 346 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 178 178 178 354 354 354 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 176 176 176 350 350 350 F F F 15 300-319 177 177 177 351 351 351 F F F 16 320-339 178 178 178 355 355 355 F F F 17 340-359 178 178 178 355 355 355 F F F 18 360-379 178 178 178 352 352 352 T T T 19 380-399 177 177 177 351 351 351 F F F 20 400-419 177 177 177 351 351 351 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 177 177 177 351 351 351 F F F 23 460-479 177 177 177 351 351 351 T T T 24 480-499 178 178 178 355 355 355 F F F 25 500-519 177 177 177 351 351 351 F F F 26 520-539 178 178 178 355 355 355 F F F 27 540-559 178 178 178 354 354 354 F F F 28 560-579 178 178 178 353 353 353 F F F 29 580-599 175 175 175 349 349 349 F F F Max gen: 178, Max flush: 355 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 355 355 355 F F F 1 020-039 178 178 178 355 355 355 F F F 2 040-059 177 177 177 351 351 351 F F F 3 060-079 177 177 177 351 351 351 F F F 4 080-099 175 175 175 349 349 349 F F F 5 100-119 176 176 176 350 350 350 F F F 6 120-139 178 178 178 354 354 354 F F F 7 140-159 178 178 178 352 352 352 F F F 8 160-179 178 178 178 352 352 352 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 174 174 174 346 346 346 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 178 178 178 354 354 354 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 176 176 176 350 350 350 F F F 15 300-319 177 177 177 351 351 351 F F F 16 320-339 178 178 178 355 355 355 F F F 17 340-359 178 178 178 355 355 355 F F F 18 360-379 179 179 179 356 356 356 F F F 19 380-399 177 177 177 351 351 351 F F F 20 400-419 177 177 177 351 351 351 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 177 177 177 351 351 351 F F F 23 460-479 179 179 179 356 356 356 F F F 24 480-499 178 178 178 355 355 355 F F F 25 500-519 177 177 177 351 351 351 F F F 26 520-539 178 178 178 355 355 355 F F F 27 540-559 178 178 178 354 354 354 F F F 28 560-579 178 178 178 353 353 353 F F F 29 580-599 175 175 175 349 349 349 F F F Max gen: 179, Max flush: 356 Begin loop 90 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.813974535Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.814283025Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.814301273Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905} {"msg":"Crucible f0c2eb72-f50b-409f-8220-a0fc7f983c62 has session id: b9e3c1db-381e-4948-a772-788cb2fed4e8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.814778415Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Upstairs opts: Upstairs UUID: f0c2eb72-f50b-409f-8220-a0fc7f983c62, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.814812072Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Crucible stats registered with UUID: f0c2eb72-f50b-409f-8220-a0fc7f983c62","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.814827931Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815160098Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905} {"msg":"f0c2eb72-f50b-409f-8220-a0fc7f983c62 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815197064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815211533Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"0","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815225961Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"0","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.81523943Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"1","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815253188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"1","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815267607Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815281175Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815345649Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"io task","client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815364667Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"io task","client":"0","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815378726Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"io task","client":"1","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"ds_connection connected from Ok(127.0.0.1:62454)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.81553784Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"io task","client":"0","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"ds_connection connected from Ok(127.0.0.1:44538)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.81564146Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"io task","client":"1","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:57.815834961Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"io task","client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:57.815864628Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"io task","client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:58:57.815885156Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815904804Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.815928941Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.816106104Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"io task","client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.816365258Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"0","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.816391205Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"0","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.816405454Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"1","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.816419343Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"1","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.817004285Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"1","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.817128312Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"0","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.817482427Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"1","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.817542331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"1","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.817568209Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.817674198Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"0","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.817701926Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"0","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:58:57.817718724Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.818089744Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"io task","client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"ds_connection connected from Ok(127.0.0.1:43452)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.818270626Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"io task","client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.819290305Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.819317412Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.819862258Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820400845Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820428542Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[0]R flush_numbers[0..12]: [355, 355, 351, 351, 349, 350, 354, 352, 352, 350, 346, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820447Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[0]R generation[0..12]: [178, 178, 177, 177, 175, 176, 178, 178, 178, 176, 174, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820465009Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820484227Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[1]R flush_numbers[0..12]: [355, 355, 351, 351, 349, 350, 354, 352, 352, 350, 346, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820501825Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[1]R generation[0..12]: [178, 178, 177, 177, 175, 176, 178, 178, 178, 176, 174, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820519053Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820536722Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[2]R flush_numbers[0..12]: [355, 355, 351, 351, 349, 350, 354, 352, 352, 350, 346, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.82055407Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[2]R generation[0..12]: [178, 178, 177, 177, 175, 176, 178, 178, 178, 176, 174, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820571368Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T21:59:07.820593166Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Max found gen is 180","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820613464Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Generation requested: 180 >= found:180","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820630502Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Next flush: 357","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820646701Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820663589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"":"downstairs","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820681177Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820702795Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"0","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820727163Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"1","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820742621Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"client":"2","":"downstairs","session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.82075607Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820773798Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"f0c2eb72-f50b-409f-8220-a0fc7f983c62 is now active with session: b9e3c1db-381e-4948-a772-788cb2fed4e8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820797806Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820819654Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905,"session_id":"b9e3c1db-381e-4948-a772-788cb2fed4e8"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:07.820840981Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 129 len 8192 data: 11 10 02/30 Write block 469 len 16384 data: 12 12 10 11 03/30 Read block 266 len 4096 04/30 Read block 241 len 40960 05/30 Write block 173 len 16384 data: 6 7 14 12 06/30 Write block 351 len 40960 data: 11 9 12 9 10 14 17 21 21 19 07/30 Read block 85 len 24576 08/30 Write block 284 len 20480 data: 16 16 14 11 14 09/30 Read block 446 len 24576 10/30 Write block 204 len 16384 data: 19 22 23 22 11/30 Flush 12/30 Write block 348 len 32768 data: 11 11 12 12 10 13 10 11 13/30 Read block 111 len 20480 14/30 Write block 20 len 40960 data: 11 14 12 11 12 12 13 14 17 19 15/30 Read block 374 len 32768 16/30 Read block 390 len 28672 17/30 Write block 567 len 40960 data: 9 15 14 17 22 20 20 19 16 18 18/30 Read block 41 len 40960 19/30 Read block 348 len 40960 20/30 Read block 145 len 24576 21/30 Read block 259 len 32768 22/30 Flush 23/30 Write block 470 len 40960 data: 13 11 12 10 8 6 7 7 8 8 24/30 Write block 2 len 8192 data: 4 5 25/30 Read block 213 len 20480 26/30 Read block 370 len 40960 27/30 Read block 382 len 12288 28/30 Write block 525 len 16384 data: 11 15 17 17 29/30 Write block 296 len 12288 data: 11 12 14 30/30 Read block 159 len 12288 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 10 Done Done Done false 25 Acked 1024 Write 2 Done Done Done false 26 Acked 1025 Read 5 Done Done Done false 27 Acked 1026 Read 10 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Write 4 Done Done Done false 30 Acked 1029 Write 3 Done Done Sent false 31 Acked 1030 Read 3 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 2 3 Done 7 8 6 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:59:07.848020178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1905} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 355 355 355 T T T 1 020-039 180 180 180 358 358 358 F F F 2 040-059 177 177 177 351 351 351 F F F 3 060-079 177 177 177 351 351 351 F F F 4 080-099 175 175 175 349 349 349 F F F 5 100-119 176 176 176 350 350 350 F F F 6 120-139 180 180 180 357 357 357 F F F 7 140-159 178 178 178 352 352 352 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 180 180 180 357 357 357 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 178 178 178 354 354 354 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 180 180 180 357 357 357 T T T 15 300-319 177 177 177 351 351 351 F F F 16 320-339 178 178 178 355 355 355 F F F 17 340-359 180 180 180 358 358 358 F F F 18 360-379 180 180 180 357 357 357 F F F 19 380-399 177 177 177 351 351 351 F F F 20 400-419 177 177 177 351 351 351 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 177 177 177 351 351 351 F F F 23 460-479 180 180 180 357 357 357 T T T 24 480-499 178 178 178 355 355 355 F F F 25 500-519 177 177 177 351 351 351 F F F 26 520-539 178 178 178 355 355 355 T T T 27 540-559 178 178 178 354 354 354 F F F 28 560-579 180 180 180 358 358 358 F F F 29 580-599 175 175 175 349 349 349 F F F Max gen: 180, Max flush: 358 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 181 181 181 359 359 359 F F F 1 020-039 180 180 180 358 358 358 F F F 2 040-059 177 177 177 351 351 351 F F F 3 060-079 177 177 177 351 351 351 F F F 4 080-099 175 175 175 349 349 349 F F F 5 100-119 176 176 176 350 350 350 F F F 6 120-139 180 180 180 357 357 357 F F F 7 140-159 178 178 178 352 352 352 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 180 180 180 357 357 357 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 178 178 178 354 354 354 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 181 181 181 359 359 359 F F F 15 300-319 177 177 177 351 351 351 F F F 16 320-339 178 178 178 355 355 355 F F F 17 340-359 180 180 180 358 358 358 F F F 18 360-379 180 180 180 357 357 357 F F F 19 380-399 177 177 177 351 351 351 F F F 20 400-419 177 177 177 351 351 351 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 177 177 177 351 351 351 F F F 23 460-479 181 181 181 359 359 359 F F F 24 480-499 178 178 178 355 355 355 F F F 25 500-519 177 177 177 351 351 351 F F F 26 520-539 181 181 181 359 359 359 F F F 27 540-559 178 178 178 354 354 354 F F F 28 560-579 180 180 180 358 358 358 F F F 29 580-599 175 175 175 349 349 349 F F F Max gen: 181, Max flush: 359 Begin loop 91 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.059486751Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.059873293Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.05989734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913} {"msg":"Crucible 7c447dc3-668a-4489-9a7d-ad84f1b99791 has session id: 2772ec49-ecad-40c1-b073-881557d414f3","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060132627Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Upstairs opts: Upstairs UUID: 7c447dc3-668a-4489-9a7d-ad84f1b99791, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060178453Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Crucible stats registered with UUID: 7c447dc3-668a-4489-9a7d-ad84f1b99791","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060195341Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060494251Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913} {"msg":"7c447dc3-668a-4489-9a7d-ad84f1b99791 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060519899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060537597Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"0","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060552056Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"0","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060566054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"1","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060579923Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"1","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060595911Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060619949Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060698951Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"io task","client":"0","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060724889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"io task","client":"1","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060749096Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"io task","client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:59346)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060947906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"io task","client":"1","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:35693)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.060983393Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"io task","client":"0","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:59:22.061061635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"io task","client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:59:22.061143707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"io task","client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:59:22.061177224Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.061192392Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.061206551Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.061367965Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"io task","client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.061892253Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"0","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.06192541Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"0","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.061952277Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"1","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.061978464Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"1","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.062488374Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"1","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.062610372Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"0","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.062995373Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"1","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.063025581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"1","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.063040649Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.063100213Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"0","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.063119301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"0","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:22.06313298Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.062048612Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"io task","client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:37054)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.062123495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"io task","client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.063172531Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.063206107Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.063823476Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064372262Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064399059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[0]R flush_numbers[0..12]: [359, 358, 351, 351, 349, 350, 357, 352, 357, 350, 357, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064415267Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[0]R generation[0..12]: [181, 180, 177, 177, 175, 176, 180, 178, 180, 176, 180, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064430286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064444884Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[1]R flush_numbers[0..12]: [359, 358, 351, 351, 349, 350, 357, 352, 357, 350, 357, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064459193Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[1]R generation[0..12]: [181, 180, 177, 177, 175, 176, 180, 178, 180, 176, 180, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064473272Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.06448786Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[2]R flush_numbers[0..12]: [359, 358, 351, 351, 349, 350, 357, 352, 357, 350, 357, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064502569Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[2]R generation[0..12]: [181, 180, 177, 177, 175, 176, 180, 178, 180, 176, 180, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064526487Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064545005Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"Wait for a query_work_queue command to finish before sending IO msg":"Max found gen is 182","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064565293Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064582171Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Next flush: 360","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064598599Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064614858Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"":"downstairs","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064639565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064665963Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"0","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064698059Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"1","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064727716Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"client":"2","":"downstairs","session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064747065Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064774172Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"7c447dc3-668a-4489-9a7d-ad84f1b99791 is now active with session: 2772ec49-ecad-40c1-b073-881557d414f3","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064803789Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064831976Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913,"session_id":"2772ec49-ecad-40c1-b073-881557d414f3"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:32.064849735Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 293 len 12288 data: 11 11 12 02/30 Read block 549 len 24576 03/30 Write block 12 len 20480 data: 17 17 18 16 16 04/30 Read block 194 len 36864 05/30 Read block 97 len 4096 06/30 Write block 470 len 4096 data: 14 07/30 Read block 135 len 40960 08/30 Read block 253 len 36864 09/30 Write block 213 len 20480 data: 16 15 16 15 20 10/30 Write block 339 len 4096 data: 12 11/30 Read block 47 len 4096 12/30 Write block 508 len 36864 data: 15 12 13 13 15 13 13 12 10 13/30 Flush 14/30 Write block 150 len 24576 data: 11 13 11 10 11 11 15/30 Read block 431 len 4096 16/30 Read block 21 len 28672 17/30 Read block 469 len 16384 18/30 Read block 52 len 28672 19/30 Read block 482 len 24576 20/30 Read block 134 len 40960 21/30 Read block 197 len 4096 22/30 Read block 504 len 36864 23/30 Read block 310 len 16384 24/30 Write block 321 len 32768 data: 19 16 15 13 13 12 12 11 25/30 Flush 26/30 Write block 545 len 12288 data: 12 16 17 27/30 Read block 460 len 8192 28/30 Read block 412 len 36864 29/30 Write block 554 len 16384 data: 17 18 18 18 30/30 Read block 36 len 16384 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 3 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Write 4 Done Done Done false 31 Acked 1030 Read 4 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 0 2 Done 4 4 5 13 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:59:32.092598244Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1913} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 360 360 360 F F F 1 020-039 180 180 180 358 358 358 F F F 2 040-059 177 177 177 351 351 351 F F F 3 060-079 177 177 177 351 351 351 F F F 4 080-099 175 175 175 349 349 349 F F F 5 100-119 176 176 176 350 350 350 F F F 6 120-139 180 180 180 357 357 357 F F F 7 140-159 182 182 182 361 361 361 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 182 182 182 360 360 360 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 178 178 178 354 354 354 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 182 182 182 360 360 360 F F F 15 300-319 177 177 177 351 351 351 F F F 16 320-339 182 182 182 362 362 362 F F F 17 340-359 180 180 180 358 358 358 F F F 18 360-379 180 180 180 357 357 357 F F F 19 380-399 177 177 177 351 351 351 F F F 20 400-419 177 177 177 351 351 351 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 177 177 177 351 351 351 F F F 23 460-479 182 182 182 360 360 360 F F F 24 480-499 178 178 178 355 355 355 F F F 25 500-519 182 182 182 360 360 360 F F F 26 520-539 181 181 181 359 359 359 F F F 27 540-559 178 178 178 354 354 354 T T T 28 560-579 180 180 180 358 358 358 F F F 29 580-599 175 175 175 349 349 349 F F F Max gen: 182, Max flush: 362 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 360 360 360 F F F 1 020-039 180 180 180 358 358 358 F F F 2 040-059 177 177 177 351 351 351 F F F 3 060-079 177 177 177 351 351 351 F F F 4 080-099 175 175 175 349 349 349 F F F 5 100-119 176 176 176 350 350 350 F F F 6 120-139 180 180 180 357 357 357 F F F 7 140-159 182 182 182 361 361 361 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 182 182 182 360 360 360 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 178 178 178 354 354 354 F F F 13 260-279 172 172 172 344 344 344 F F F 14 280-299 182 182 182 360 360 360 F F F 15 300-319 177 177 177 351 351 351 F F F 16 320-339 182 182 182 362 362 362 F F F 17 340-359 180 180 180 358 358 358 F F F 18 360-379 180 180 180 357 357 357 F F F 19 380-399 177 177 177 351 351 351 F F F 20 400-419 177 177 177 351 351 351 F F F 21 420-439 168 168 168 334 334 334 F F F 22 440-459 177 177 177 351 351 351 F F F 23 460-479 182 182 182 360 360 360 F F F 24 480-499 178 178 178 355 355 355 F F F 25 500-519 182 182 182 360 360 360 F F F 26 520-539 181 181 181 359 359 359 F F F 27 540-559 183 183 183 363 363 363 F F F 28 560-579 180 180 180 358 358 358 F F F 29 580-599 175 175 175 349 349 349 F F F Max gen: 183, Max flush: 363 Begin loop 92 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.266077165Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.266390824Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.266409752Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921} {"msg":"Crucible dbdb408d-10a3-4c38-bc6c-ad6309625ff2 has session id: 9f387e2f-a051-44cb-b7bb-950e5239f090","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.266748878Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Upstairs opts: Upstairs UUID: dbdb408d-10a3-4c38-bc6c-ad6309625ff2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.266797523Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Crucible stats registered with UUID: dbdb408d-10a3-4c38-bc6c-ad6309625ff2","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.266821491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267167877Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921} {"msg":"dbdb408d-10a3-4c38-bc6c-ad6309625ff2 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267204013Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267228461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"0","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267251889Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"0","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267274946Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"1","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267306773Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"1","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267328641Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267350169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267370657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"io task","client":"0","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267392704Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"io task","client":"1","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267414772Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"io task","client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"ds_connection connected from Ok(127.0.0.1:45113)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.26743736Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"io task","client":"0","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"ds_connection connected from Ok(127.0.0.1:58894)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267459698Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"io task","client":"1","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:59:46.267578916Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"io task","client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:59:46.267620692Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"io task","client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:59:46.267652749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267680856Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267700094Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.267783586Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"io task","client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.268237451Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"0","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.268265288Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"0","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.268280876Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"1","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.268295335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"1","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.268774847Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"1","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.268879057Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"0","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.269233962Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"1","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.269261909Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"1","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.269276648Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.269289256Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"0","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.269302985Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"0","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:46.269316174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.268689171Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"io task","client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"ds_connection connected from Ok(127.0.0.1:55054)","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.26890272Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"io task","client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.269978503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.270020759Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.27071849Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271286024Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271312701Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[0]R flush_numbers[0..12]: [360, 358, 351, 351, 349, 350, 357, 361, 357, 350, 360, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271329049Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[0]R generation[0..12]: [182, 180, 177, 177, 175, 176, 180, 182, 180, 176, 182, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271344678Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271359397Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[1]R flush_numbers[0..12]: [360, 358, 351, 351, 349, 350, 357, 361, 357, 350, 360, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271373585Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[1]R generation[0..12]: [182, 180, 177, 177, 175, 176, 180, 182, 180, 176, 182, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271387424Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271401682Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[2]R flush_numbers[0..12]: [360, 358, 351, 351, 349, 350, 357, 361, 357, 350, 360, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271416371Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[2]R generation[0..12]: [182, 180, 177, 177, 175, 176, 180, 182, 180, 176, 182, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271434259Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271448808Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271461876Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271478095Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Next flush: 364","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271491193Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271504302Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"":"downstairs","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271517501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271529589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"0","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271543158Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"1","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271556297Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"client":"2","":"downstairs","session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271569506Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271582074Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"dbdb408d-10a3-4c38-bc6c-ad6309625ff2 is now active with session: 9f387e2f-a051-44cb-b7bb-950e5239f090","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271595023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271607262Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921,"session_id":"9f387e2f-a051-44cb-b7bb-950e5239f090"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2024-07-09T21:59:56.271619491Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 366 len 40960 data: 17 15 15 20 17 14 17 18 18 17 02/30 Read block 46 len 8192 03/30 Write block 386 len 8192 data: 22 22 04/30 Read block 292 len 36864 05/30 Flush 06/30 Write block 69 len 20480 data: 13 15 16 14 12 07/30 Read block 585 len 16384 08/30 Read block 335 len 40960 09/30 Read block 386 len 20480 10/30 Read block 199 len 24576 11/30 Read block 395 len 16384 12/30 Read block 438 len 36864 13/30 Write block 458 len 20480 data: 13 11 10 10 14 14/30 Read block 225 len 12288 15/30 Write block 419 len 36864 data: 13 15 14 14 12 11 12 12 14 16/30 Read block 126 len 12288 17/30 Write block 445 len 36864 data: 16 13 14 16 18 17 18 18 17 18/30 Write block 471 len 8192 data: 12 13 19/30 Read block 119 len 24576 20/30 Write block 349 len 16384 data: 12 13 13 11 21/30 Read block 102 len 24576 22/30 Read block 217 len 4096 23/30 Write block 309 len 16384 data: 14 14 12 14 24/30 Read block 73 len 24576 25/30 Write block 306 len 4096 data: 16 26/30 Write block 273 len 36864 data: 10 9 11 9 9 9 11 13 12 27/30 Write block 509 len 32768 data: 13 14 14 16 14 14 13 11 28/30 Write block 133 len 28672 data: 14 15 12 10 13 13 13 29/30 Write block 1 len 20480 data: 4 5 6 6 7 30/30 Write block 514 len 16384 data: 15 14 12 14 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 21 Acked 1020 Write 4 Done Done Done false 22 Acked 1021 Read 6 Done Done Done false 23 Acked 1022 Read 1 Done Done Done false 24 Acked 1023 Write 4 Done Done Done false 25 Acked 1024 Read 6 Done Done Done false 26 Acked 1025 Write 1 Done Done Done false 27 Acked 1026 Write 9 Done Done Done false 28 Acked 1027 Write 8 Done Done Done false 29 Acked 1028 Write 7 Done Done Done false 30 Acked 1029 Write 5 Done Done Done false 31 Acked 1030 Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 10 10 10 30 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1019 1019 1019 Downstairs last five completed: 1019 1018 1017 1016 1015 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T21:59:56.295362238Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1921} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 360 360 360 T T T 1 020-039 180 180 180 358 358 358 F F F 2 040-059 177 177 177 351 351 351 F F F 3 060-079 184 184 184 365 365 365 F F F 4 080-099 175 175 175 349 349 349 F F F 5 100-119 176 176 176 350 350 350 F F F 6 120-139 180 180 180 357 357 357 T T T 7 140-159 182 182 182 361 361 361 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 182 182 182 360 360 360 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 178 178 178 354 354 354 F F F 13 260-279 172 172 172 344 344 344 T T T 14 280-299 182 182 182 360 360 360 T T T 15 300-319 177 177 177 351 351 351 T T T 16 320-339 182 182 182 362 362 362 F F F 17 340-359 180 180 180 358 358 358 T T T 18 360-379 184 184 184 364 364 364 F F F 19 380-399 184 184 184 364 364 364 F F F 20 400-419 184 184 184 365 365 365 F F F 21 420-439 184 184 184 365 365 365 F F F 22 440-459 184 184 184 365 365 365 F F F 23 460-479 184 184 184 365 365 365 F F F 24 480-499 178 178 178 355 355 355 F F F 25 500-519 182 182 182 360 360 360 T T T 26 520-539 181 181 181 359 359 359 F F F 27 540-559 183 183 183 363 363 363 F F F 28 560-579 180 180 180 358 358 358 F F F 29 580-599 175 175 175 349 349 349 F F F Max gen: 184, Max flush: 365 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 185 185 185 366 366 366 F F F 1 020-039 180 180 180 358 358 358 F F F 2 040-059 177 177 177 351 351 351 F F F 3 060-079 184 184 184 365 365 365 F F F 4 080-099 175 175 175 349 349 349 F F F 5 100-119 176 176 176 350 350 350 F F F 6 120-139 185 185 185 366 366 366 F F F 7 140-159 182 182 182 361 361 361 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 182 182 182 360 360 360 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 178 178 178 354 354 354 F F F 13 260-279 185 185 185 366 366 366 F F F 14 280-299 185 185 185 366 366 366 F F F 15 300-319 185 185 185 366 366 366 F F F 16 320-339 182 182 182 362 362 362 F F F 17 340-359 185 185 185 366 366 366 F F F 18 360-379 184 184 184 364 364 364 F F F 19 380-399 184 184 184 364 364 364 F F F 20 400-419 184 184 184 365 365 365 F F F 21 420-439 184 184 184 365 365 365 F F F 22 440-459 184 184 184 365 365 365 F F F 23 460-479 184 184 184 365 365 365 F F F 24 480-499 178 178 178 355 355 355 F F F 25 500-519 185 185 185 366 366 366 F F F 26 520-539 181 181 181 359 359 359 F F F 27 540-559 183 183 183 363 363 363 F F F 28 560-579 180 180 180 358 358 358 F F F 29 580-599 175 175 175 349 349 349 F F F Max gen: 185, Max flush: 366 Begin loop 93 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.544774372Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.545554965Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.545583632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929} {"msg":"Crucible 6409eb3a-0157-47df-be57-218f303d2186 has session id: de0222dd-cde5-4b30-9c46-18c7dc1d83c1","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.545613709Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Upstairs opts: Upstairs UUID: 6409eb3a-0157-47df-be57-218f303d2186, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.545756305Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Crucible stats registered with UUID: 6409eb3a-0157-47df-be57-218f303d2186","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.545786342Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546064624Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929} {"msg":"6409eb3a-0157-47df-be57-218f303d2186 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546133048Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546151506Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"0","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546166674Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"0","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546180693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546194461Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.54620794Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"2","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546229148Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"2","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546252996Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"io task","client":"2","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546268584Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"io task","client":"0","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546282363Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"io task","client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:10.546483693Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"io task","client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"ds_connection connected from Ok(127.0.0.1:39455)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.54661649Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"io task","client":"0","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:10.546653656Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"io task","client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"ds_connection connected from Ok(127.0.0.1:38519)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546684793Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"io task","client":"2","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:10.5467152Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546738258Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546764465Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.546859935Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"io task","client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.547431679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"0","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.547456966Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"0","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.547471725Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"2","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.547488373Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"2","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.548054627Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"2","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.548094083Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"0","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.548618421Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"0","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.548655598Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"0","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.548683635Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.548722041Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"2","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.548754988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"2","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:10.548771856Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.547953962Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"io task","client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"ds_connection connected from Ok(127.0.0.1:55548)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.548045403Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"io task","client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.549114627Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.549138165Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.549831986Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.55050286Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550537556Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[0]R flush_numbers[0..12]: [366, 358, 351, 365, 349, 350, 366, 361, 357, 350, 360, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550558274Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[0]R generation[0..12]: [185, 180, 177, 184, 175, 176, 185, 182, 180, 176, 182, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550577042Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.55059539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[1]R flush_numbers[0..12]: [366, 358, 351, 365, 349, 350, 366, 361, 357, 350, 360, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550613289Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[1]R generation[0..12]: [185, 180, 177, 184, 175, 176, 185, 182, 180, 176, 182, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550630757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550648435Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"Wait for a query_work_queue command to finish before sending IO msg":"[2]R flush_numbers[0..12]: [366, 358, 351, 365, 349, 350, 366, 361, 357, 350, 360, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550674223Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[2]R generation[0..12]: [185, 180, 177, 184, 175, 176, 185, 182, 180, 176, 182, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550692331Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550710269Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Max found gen is 186","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550726847Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Generation requested: 186 >= found:186","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550743656Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Next flush: 367","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550760084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550776332Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"":"downstairs","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550792901Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550808099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"0","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550824877Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"1","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550841456Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"client":"2","":"downstairs","session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550858214Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550873593Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"6409eb3a-0157-47df-be57-218f303d2186 is now active with session: de0222dd-cde5-4b30-9c46-18c7dc1d83c1","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550889921Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.5509051Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929,"session_id":"de0222dd-cde5-4b30-9c46-18c7dc1d83c1"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:20.550922688Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 585 len 40960 data: 12 13 17 16 18 18 19 17 19 18 02/30 Flush 03/30 Write block 24 len 32768 data: 13 13 14 15 18 20 18 17 04/30 Write block 436 len 12288 data: 13 13 13 05/30 Write block 326 len 40960 data: 13 13 12 11 12 13 12 11 11 9 06/30 Write block 69 len 40960 data: 14 16 17 15 13 14 11 12 13 14 07/30 Write block 253 len 4096 data: 9 08/30 Write block 154 len 16384 data: 12 12 10 8 09/30 Flush 10/30 Write block 16 len 32768 data: 17 16 12 10 12 15 13 12 11/30 Write block 339 len 20480 data: 13 12 13 14 15 12/30 Write block 363 len 36864 data: 13 13 15 18 16 16 21 18 15 13/30 Write block 488 len 36864 data: 19 18 17 16 9 9 9 8 6 14/30 Write block 422 len 4096 data: 15 15/30 Read block 32 len 20480 16/30 Read block 177 len 8192 17/30 Write block 65 len 40960 data: 15 16 13 14 15 17 18 16 14 15 18/30 Read block 103 len 36864 19/30 Read block 14 len 24576 20/30 Read block 579 len 24576 21/30 Write block 96 len 32768 data: 9 12 12 13 15 14 13 17 22/30 Read block 512 len 16384 23/30 Write block 464 len 16384 data: 10 8 8 9 24/30 Read block 198 len 24576 25/30 Write block 312 len 20480 data: 15 13 14 11 14 26/30 Read block 400 len 28672 27/30 Write block 58 len 24576 data: 16 15 14 16 18 15 28/30 Write block 19 len 24576 data: 11 13 16 14 13 14 29/30 Write block 440 len 12288 data: 14 16 16 30/30 Read block 58 len 20480 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 10 Done Done Done false 19 Acked 1018 Read 9 Done Done Done false 20 Acked 1019 Read 6 Done Done Done false 21 Acked 1020 Read 6 Done Done Done false 22 Acked 1021 Write 8 Done Done Done false 23 Acked 1022 Read 4 Done Done Done false 24 Acked 1023 Write 4 Done Done Done false 25 Acked 1024 Read 6 Done Done Done false 26 Acked 1025 Write 5 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Write 6 Done Done Done false 29 Acked 1028 Write 6 Done Done Done false 30 Acked 1029 Write 3 Done Done Done false 31 Acked 1030 Read 5 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 14 14 14 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:20.578535551Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1929} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 186 186 186 369 369 369 T T T 1 020-039 186 186 186 369 369 369 T T T 2 040-059 177 177 177 351 351 351 T T T 3 060-079 186 186 186 368 368 368 T T T 4 080-099 175 175 175 349 349 349 T T T 5 100-119 176 176 176 350 350 350 T T T 6 120-139 185 185 185 366 366 366 F F F 7 140-159 186 186 186 368 368 368 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 182 182 182 360 360 360 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 186 186 186 368 368 368 F F F 13 260-279 185 185 185 366 366 366 F F F 14 280-299 185 185 185 366 366 366 F F F 15 300-319 185 185 185 366 366 366 T T T 16 320-339 186 186 186 369 369 369 F F F 17 340-359 186 186 186 369 369 369 F F F 18 360-379 186 186 186 369 369 369 F F F 19 380-399 184 184 184 364 364 364 F F F 20 400-419 184 184 184 365 365 365 F F F 21 420-439 186 186 186 369 369 369 F F F 22 440-459 184 184 184 365 365 365 T T T 23 460-479 184 184 184 365 365 365 T T T 24 480-499 186 186 186 369 369 369 F F F 25 500-519 185 185 185 366 366 366 F F F 26 520-539 181 181 181 359 359 359 F F F 27 540-559 183 183 183 363 363 363 F F F 28 560-579 180 180 180 358 358 358 F F F 29 580-599 186 186 186 367 367 367 F F F Max gen: 186, Max flush: 369 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 187 187 187 370 370 370 F F F 2 040-059 187 187 187 370 370 370 F F F 3 060-079 187 187 187 370 370 370 F F F 4 080-099 187 187 187 370 370 370 F F F 5 100-119 187 187 187 370 370 370 F F F 6 120-139 185 185 185 366 366 366 F F F 7 140-159 186 186 186 368 368 368 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 176 176 176 350 350 350 F F F 10 200-219 182 182 182 360 360 360 F F F 11 220-239 175 175 175 349 349 349 F F F 12 240-259 186 186 186 368 368 368 F F F 13 260-279 185 185 185 366 366 366 F F F 14 280-299 185 185 185 366 366 366 F F F 15 300-319 187 187 187 370 370 370 F F F 16 320-339 186 186 186 369 369 369 F F F 17 340-359 186 186 186 369 369 369 F F F 18 360-379 186 186 186 369 369 369 F F F 19 380-399 184 184 184 364 364 364 F F F 20 400-419 184 184 184 365 365 365 F F F 21 420-439 186 186 186 369 369 369 F F F 22 440-459 187 187 187 370 370 370 F F F 23 460-479 187 187 187 370 370 370 F F F 24 480-499 186 186 186 369 369 369 F F F 25 500-519 185 185 185 366 366 366 F F F 26 520-539 181 181 181 359 359 359 F F F 27 540-559 183 183 183 363 363 363 F F F 28 560-579 180 180 180 358 358 358 F F F 29 580-599 186 186 186 367 367 367 F F F Max gen: 187, Max flush: 370 Begin loop 94 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.866296554Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.866563318Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.866584406Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937} {"msg":"Crucible 5784bdae-bf07-46a1-bc4f-e8f1a3786c23 has session id: f5a1c932-8765-4676-805d-ad67c26cf466","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.866995235Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Upstairs opts: Upstairs UUID: 5784bdae-bf07-46a1-bc4f-e8f1a3786c23, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.86704487Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Crucible stats registered with UUID: 5784bdae-bf07-46a1-bc4f-e8f1a3786c23","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867062278Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867464068Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937} {"msg":"5784bdae-bf07-46a1-bc4f-e8f1a3786c23 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867495495Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867518143Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867541031Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867564198Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"1","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867586906Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"1","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867609704Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"2","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867635231Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"2","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867661109Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"io task","client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867688816Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"io task","client":"1","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867714514Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"io task","client":"2","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"ds_connection connected from Ok(127.0.0.1:33561)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867818063Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"io task","client":"2","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"ds_connection connected from Ok(127.0.0.1:47269)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.867915894Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"io task","client":"1","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:34.868037642Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"io task","client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:34.868074598Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"io task","client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:34.868146711Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.868165619Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.868179728Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.868315024Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"io task","client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.868714755Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"1","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.868751791Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"1","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.868781718Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"2","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.868809135Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"2","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.869333683Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"1","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.86946775Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"2","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.869800487Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"1","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.869826834Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"1","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.869841283Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.869923625Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"2","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.869946022Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"2","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:34.869962071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.86892355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"io task","client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"ds_connection connected from Ok(127.0.0.1:64727)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.869102652Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"io task","client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.8700375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.870072246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.870688765Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871255969Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871283396Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[0]R flush_numbers[0..12]: [370, 370, 370, 370, 370, 370, 366, 368, 357, 350, 360, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871300044Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[0]R generation[0..12]: [187, 187, 187, 187, 187, 187, 185, 186, 180, 176, 182, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871315113Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871329992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[1]R flush_numbers[0..12]: [370, 370, 370, 370, 370, 370, 366, 368, 357, 350, 360, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.87134778Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[1]R generation[0..12]: [187, 187, 187, 187, 187, 187, 185, 186, 180, 176, 182, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871362788Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871377107Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[2]R flush_numbers[0..12]: [370, 370, 370, 370, 370, 370, 366, 368, 357, 350, 360, 349]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871391075Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[2]R generation[0..12]: [187, 187, 187, 187, 187, 187, 185, 186, 180, 176, 182, 175]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871405174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871419453Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Max found gen is 188","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T22:00:44.871432641Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.87144968Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Next flush: 371","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871462978Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871476177Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"":"downstairs","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871489726Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871501775Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"0","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871515213Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"1","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871528572Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"client":"2","":"downstairs","session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.87154214Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871556639Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"5784bdae-bf07-46a1-bc4f-e8f1a3786c23 is now active with session: f5a1c932-8765-4676-805d-ad67c26cf466","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871569988Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871582227Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937,"session_id":"f5a1c932-8765-4676-805d-ad67c26cf466"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:44.871594575Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 254 len 20480 data: 8 12 12 11 10 02/30 Flush 03/30 Write block 370 len 12288 data: 19 16 18 04/30 Flush 05/30 Write block 514 len 40960 data: 16 15 13 15 13 12 14 10 8 10 06/30 Read block 248 len 20480 07/30 Flush 08/30 Write block 211 len 40960 data: 17 17 17 16 17 16 21 18 18 15 09/30 Read block 112 len 4096 10/30 Read block 526 len 20480 11/30 Flush 12/30 Write block 557 len 12288 data: 19 16 12 13/30 Read block 348 len 36864 14/30 Write block 230 len 24576 data: 10 11 10 11 12 11 15/30 Write block 470 len 12288 data: 15 13 14 16/30 Write block 582 len 24576 data: 14 12 14 13 14 18 17/30 Read block 136 len 36864 18/30 Write block 396 len 32768 data: 15 12 10 14 15 10 9 13 19/30 Read block 538 len 36864 20/30 Read block 390 len 28672 21/30 Write block 261 len 16384 data: 7 10 13 11 22/30 Read block 22 len 32768 23/30 Flush 24/30 Write block 31 len 4096 data: 18 25/30 Write block 71 len 24576 data: 19 17 15 16 12 13 26/30 Flush 27/30 Write block 291 len 40960 data: 15 14 12 12 13 12 13 15 14 16 28/30 Read block 466 len 28672 29/30 Write block 196 len 16384 data: 11 13 15 15 30/30 Write block 551 len 8192 data: 16 18 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 10 Done Done Done false 28 Acked 1027 Read 7 Done Done Done false 29 Acked 1028 Write 4 Sent Sent Sent false 30 Acked 1029 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 2 2 6 Done 2 2 2 6 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:44.906626093Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1937} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 188 188 188 376 376 376 F F F 2 040-059 187 187 187 370 370 370 F F F 3 060-079 188 188 188 376 376 376 F F F 4 080-099 187 187 187 370 370 370 F F F 5 100-119 187 187 187 370 370 370 F F F 6 120-139 185 185 185 366 366 366 F F F 7 140-159 186 186 186 368 368 368 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 176 176 176 350 350 350 T T T 10 200-219 188 188 188 374 374 374 F F F 11 220-239 188 188 188 375 375 375 F F F 12 240-259 188 188 188 371 371 371 F F F 13 260-279 188 188 188 375 375 375 F F F 14 280-299 185 185 185 366 366 366 T T T 15 300-319 187 187 187 370 370 370 T T T 16 320-339 186 186 186 369 369 369 F F F 17 340-359 186 186 186 369 369 369 F F F 18 360-379 188 188 188 372 372 372 F F F 19 380-399 188 188 188 375 375 375 F F F 20 400-419 188 188 188 375 375 375 F F F 21 420-439 186 186 186 369 369 369 F F F 22 440-459 187 187 187 370 370 370 F F F 23 460-479 188 188 188 375 375 375 F F F 24 480-499 186 186 186 369 369 369 F F F 25 500-519 188 188 188 373 373 373 F F F 26 520-539 188 188 188 373 373 373 F F F 27 540-559 188 188 188 375 375 375 T T T 28 560-579 180 180 180 358 358 358 F F F 29 580-599 188 188 188 375 375 375 F F F Max gen: 188, Max flush: 376 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 188 188 188 376 376 376 F F F 2 040-059 187 187 187 370 370 370 F F F 3 060-079 188 188 188 376 376 376 F F F 4 080-099 187 187 187 370 370 370 F F F 5 100-119 187 187 187 370 370 370 F F F 6 120-139 185 185 185 366 366 366 F F F 7 140-159 186 186 186 368 368 368 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 189 189 189 377 377 377 F F F 10 200-219 188 188 188 374 374 374 F F F 11 220-239 188 188 188 375 375 375 F F F 12 240-259 188 188 188 371 371 371 F F F 13 260-279 188 188 188 375 375 375 F F F 14 280-299 189 189 189 377 377 377 F F F 15 300-319 189 189 189 377 377 377 F F F 16 320-339 186 186 186 369 369 369 F F F 17 340-359 186 186 186 369 369 369 F F F 18 360-379 188 188 188 372 372 372 F F F 19 380-399 188 188 188 375 375 375 F F F 20 400-419 188 188 188 375 375 375 F F F 21 420-439 186 186 186 369 369 369 F F F 22 440-459 187 187 187 370 370 370 F F F 23 460-479 188 188 188 375 375 375 F F F 24 480-499 186 186 186 369 369 369 F F F 25 500-519 188 188 188 373 373 373 F F F 26 520-539 188 188 188 373 373 373 F F F 27 540-559 189 189 189 377 377 377 F F F 28 560-579 180 180 180 358 358 358 F F F 29 580-599 188 188 188 375 375 375 F F F Max gen: 189, Max flush: 377 Begin loop 95 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.124283363Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.124695773Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.1247219Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945} {"msg":"Crucible 15f82603-7dd6-4194-a272-0d0af3f0deb7 has session id: 12b3f8cd-14e8-4b62-903a-012a86653e38","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125057567Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Upstairs opts: Upstairs UUID: 15f82603-7dd6-4194-a272-0d0af3f0deb7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125105992Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Crucible stats registered with UUID: 15f82603-7dd6-4194-a272-0d0af3f0deb7","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125133919Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125409882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945} {"msg":"15f82603-7dd6-4194-a272-0d0af3f0deb7 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125483475Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125502753Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"0","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125517711Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"0","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125532Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125546038Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125562837Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"2","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125587654Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"2","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125620041Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"io task","client":"0","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125646218Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"io task","client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125676955Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"io task","client":"2","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"ds_connection connected from Ok(127.0.0.1:59580)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.125964367Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"io task","client":"0","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"ds_connection connected from Ok(127.0.0.1:42414)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.126100133Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"io task","client":"2","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:59.126150258Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"io task","client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:59.126181295Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"io task","client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:00:59.126211192Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.1262324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.126258338Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.126383815Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"io task","client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.126686715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"2","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.126713213Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"2","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.126803784Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"0","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.126823262Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"0","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.12724751Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"2","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.127373067Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"0","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.127795715Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"2","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.127833851Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"2","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.127857369Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.127877847Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"0","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.127899865Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"0","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:00:59.127921763Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.126919899Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"io task","client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"ds_connection connected from Ok(127.0.0.1:54825)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.127059675Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"io task","client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.128476905Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.12852422Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129147198Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129668986Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129690115Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[0]R flush_numbers[0..12]: [370, 376, 370, 376, 370, 370, 366, 368, 357, 377, 374, 375]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129705543Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[0]R generation[0..12]: [187, 188, 187, 188, 187, 187, 185, 186, 180, 189, 188, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129720161Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.12973454Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[1]R flush_numbers[0..12]: [370, 376, 370, 376, 370, 370, 366, 368, 357, 377, 374, 375]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129753708Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[1]R generation[0..12]: [187, 188, 187, 188, 187, 187, 185, 186, 180, 189, 188, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129768177Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129782585Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[2]R flush_numbers[0..12]: [370, 376, 370, 376, 370, 370, 366, 368, 357, 377, 374, 375]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129796694Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[2]R generation[0..12]: [187, 188, 187, 188, 187, 187, 185, 186, 180, 189, 188, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129810632Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129825061Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2024-07-09T22:01:09.12983831Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129855848Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Next flush: 378","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129869077Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129882125Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"":"downstairs","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129895574Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129907853Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"0","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129921301Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"1","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.12993452Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"client":"2","":"downstairs","session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129947949Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129962317Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"15f82603-7dd6-4194-a272-0d0af3f0deb7 is now active with session: 12b3f8cd-14e8-4b62-903a-012a86653e38","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129975576Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.129987695Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945,"session_id":"12b3f8cd-14e8-4b62-903a-012a86653e38"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:09.130000074Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 535 len 8192 data: 13 13 02/30 Write block 451 len 16384 data: 19 19 18 18 03/30 Read block 98 len 40960 04/30 Write block 583 len 28672 data: 13 15 14 15 19 17 19 05/30 Read block 507 len 8192 06/30 Read block 467 len 20480 07/30 Write block 296 len 16384 data: 13 14 16 15 08/30 Read block 563 len 36864 09/30 Write block 425 len 8192 data: 13 13 10/30 Read block 537 len 36864 11/30 Write block 135 len 36864 data: 13 11 14 14 14 13 12 13 13 12/30 Read block 173 len 16384 13/30 Write block 293 len 16384 data: 13 13 14 14 14/30 Write block 269 len 32768 data: 11 8 11 10 11 10 12 10 15/30 Read block 383 len 28672 16/30 Write block 476 len 4096 data: 8 17/30 Write block 558 len 24576 data: 17 13 12 12 12 13 18/30 Read block 239 len 4096 19/30 Read block 568 len 32768 20/30 Write block 261 len 4096 data: 8 21/30 Read block 249 len 20480 22/30 Write block 374 len 28672 data: 19 18 16 11 9 12 11 23/30 Read block 470 len 28672 24/30 Read block 190 len 4096 25/30 Read block 119 len 8192 26/30 Read block 108 len 32768 27/30 Write block 139 len 40960 data: 15 14 13 14 14 13 13 14 11 9 28/30 Write block 42 len 8192 data: 11 13 29/30 Write block 462 len 20480 data: 15 14 11 9 9 30/30 Write block 435 len 28672 data: 17 14 14 14 10 15 17 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:1 downstairs:31 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1 Acked 1000 Write 2 Done Done Done false 2 Acked 1001 Write 4 Done Done Done false 3 Acked 1002 Read 10 Done Done Done false 4 Acked 1003 Write 7 Done Done Done false 5 Acked 1004 Read 2 Done Done Done false 6 Acked 1005 Read 5 Done Done Done false 7 Acked 1006 Write 4 Done Done Done false 8 Acked 1007 Read 9 Done Done Done false 9 Acked 1008 Write 2 Done Done Done false 10 Acked 1009 Read 9 Done Done Done false 11 Acked 1010 Write 9 Done Done Done false 12 Acked 1011 Read 4 Done Done Done false 13 Acked 1012 Write 4 Done Done Done false 14 Acked 1013 Write 8 Done Done Done false 15 Acked 1014 Read 7 Done Done Done false 16 Acked 1015 Write 1 Done Done Done false 17 Acked 1016 Write 6 Done Done Done false 18 Acked 1017 Read 1 Done Done Done false 19 Acked 1018 Read 8 Done Done Done false 20 Acked 1019 Write 1 Done Done Done false 21 Acked 1020 Read 5 Done Done Done false 22 Acked 1021 Write 7 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Read 1 Done Done Done false 25 Acked 1024 Read 2 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 NotAcked 1026 Flush 0 Sent Sent Sent false 28 Acked 1027 Write 10 Sent Sent Sent false 29 Acked 1028 Write 2 Sent Sent Sent false 30 Acked 1029 Write 5 Sent Sent Sent false 31 Acked 1030 Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 5 5 5 15 Done 26 26 26 78 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 0 0 Downstairs last five completed: Upstairs last five completed: 31 30 29 28 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:01:09.151541339Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1945} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 188 188 188 376 376 376 F F F 2 040-059 187 187 187 370 370 370 T T T 3 060-079 188 188 188 376 376 376 F F F 4 080-099 187 187 187 370 370 370 F F F 5 100-119 187 187 187 370 370 370 F F F 6 120-139 190 190 190 378 378 378 T T T 7 140-159 190 190 190 378 378 378 T T T 8 160-179 180 180 180 357 357 357 F F F 9 180-199 189 189 189 377 377 377 F F F 10 200-219 188 188 188 374 374 374 F F F 11 220-239 188 188 188 375 375 375 F F F 12 240-259 188 188 188 371 371 371 F F F 13 260-279 190 190 190 378 378 378 F F F 14 280-299 190 190 190 378 378 378 F F F 15 300-319 189 189 189 377 377 377 F F F 16 320-339 186 186 186 369 369 369 F F F 17 340-359 186 186 186 369 369 369 F F F 18 360-379 190 190 190 378 378 378 F F F 19 380-399 190 190 190 378 378 378 F F F 20 400-419 188 188 188 375 375 375 F F F 21 420-439 190 190 190 378 378 378 F F F 22 440-459 190 190 190 378 378 378 F F F 23 460-479 190 190 190 378 378 378 T T T 24 480-499 186 186 186 369 369 369 F F F 25 500-519 188 188 188 373 373 373 F F F 26 520-539 190 190 190 378 378 378 F F F 27 540-559 190 190 190 378 378 378 F F F 28 560-579 190 190 190 378 378 378 F F F 29 580-599 190 190 190 378 378 378 F F F Max gen: 190, Max flush: 378 A Difference in extent metadata is expected here On loop 95 Verifying data now Files /tmp/test_repair_verify.data and /tmp/test_repair_verify.data.last differ 438,442c438 < 16, < 13, < 13, < 13, < 9, --- > 17, 443a440,444 > 14, > 14, > 10, > 15, > 17, 445d445 < 16, diff found after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 188 188 188 376 376 376 F F F 2 040-059 191 191 191 379 379 379 F F F 3 060-079 188 188 188 376 376 376 F F F 4 080-099 187 187 187 370 370 370 F F F 5 100-119 187 187 187 370 370 370 F F F 6 120-139 191 191 191 379 379 379 F F F 7 140-159 191 191 191 379 379 379 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 189 189 189 377 377 377 F F F 10 200-219 188 188 188 374 374 374 F F F 11 220-239 188 188 188 375 375 375 F F F 12 240-259 188 188 188 371 371 371 F F F 13 260-279 190 190 190 378 378 378 F F F 14 280-299 190 190 190 378 378 378 F F F 15 300-319 189 189 189 377 377 377 F F F 16 320-339 186 186 186 369 369 369 F F F 17 340-359 186 186 186 369 369 369 F F F 18 360-379 190 190 190 378 378 378 F F F 19 380-399 190 190 190 378 378 378 F F F 20 400-419 188 188 188 375 375 375 F F F 21 420-439 190 190 190 378 378 378 F F F 22 440-459 190 190 190 378 378 378 F F F 23 460-479 191 191 191 379 379 379 F F F 24 480-499 186 186 186 369 369 369 F F F 25 500-519 188 188 188 373 373 373 F F F 26 520-539 190 190 190 378 378 378 F F F 27 540-559 190 190 190 378 378 378 F F F 28 560-579 190 190 190 378 378 378 F F F 29 580-599 190 190 190 378 378 378 F F F Max gen: 191, Max flush: 379 Begin loop 96 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.406089654Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.406476296Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.406505233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956} {"msg":"Crucible 62bcf86e-26cc-491d-b434-3656498f0cc6 has session id: d9cd721f-5bb7-41c5-8742-3130766bd353","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.406847349Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Upstairs opts: Upstairs UUID: 62bcf86e-26cc-491d-b434-3656498f0cc6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.406893064Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Crucible stats registered with UUID: 62bcf86e-26cc-491d-b434-3656498f0cc6","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.406916402Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407175986Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956} {"msg":"62bcf86e-26cc-491d-b434-3656498f0cc6 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407214552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407230071Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"0","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407244789Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"0","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407258498Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"1","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407275386Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"1","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407289335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407303054Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407347799Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"io task","client":"0","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407367697Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"io task","client":"1","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407385235Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"io task","client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"ds_connection connected from Ok(127.0.0.1:44680)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407605204Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"io task","client":"0","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:01:23.40764315Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"io task","client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"ds_connection connected from Ok(127.0.0.1:52341)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407670797Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"io task","client":"1","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:01:23.407697764Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"io task","client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:01:23.407725522Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407801474Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407823062Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.407932481Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"io task","client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.408404425Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"0","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.408431552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"0","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.40844656Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"1","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.408466488Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"1","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.409044911Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"0","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.40915864Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"1","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.409532862Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"0","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.40955933Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"0","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.409574178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.409643882Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"1","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.4096621Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"1","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:23.409676028Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.408698052Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"io task","client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"ds_connection connected from Ok(127.0.0.1:53680)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.408868095Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"io task","client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.410291344Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.410325001Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.410888355Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411430771Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411450289Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[0]R flush_numbers[0..12]: [370, 376, 379, 376, 370, 370, 379, 379, 357, 377, 374, 375]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411478176Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[0]R generation[0..12]: [187, 188, 191, 188, 187, 187, 191, 191, 180, 189, 188, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411492995Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411507593Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[1]R flush_numbers[0..12]: [370, 376, 379, 376, 370, 370, 379, 379, 357, 377, 374, 375]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411521752Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[1]R generation[0..12]: [187, 188, 191, 188, 187, 187, 191, 191, 180, 189, 188, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.41153592Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411550579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[2]R flush_numbers[0..12]: [370, 376, 379, 376, 370, 370, 379, 379, 357, 377, 374, 375]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411564808Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[2]R generation[0..12]: [187, 188, 191, 188, 187, 187, 191, 191, 180, 189, 188, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411578766Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411592855Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411606084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411623422Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} Wait for a query_work_queue command to finish before sending IO {"msg":"Next flush: 380","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.41164017Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411653929Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"":"downstairs","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411667607Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411679876Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"0","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411693395Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"1","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411706714Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"client":"2","":"downstairs","session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411725552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.41174823Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"62bcf86e-26cc-491d-b434-3656498f0cc6 is now active with session: d9cd721f-5bb7-41c5-8742-3130766bd353","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411772117Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411793485Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956,"session_id":"d9cd721f-5bb7-41c5-8742-3130766bd353"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:33.411814703Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 540 len 20480 data: 14 12 11 12 12 02/30 Read block 316 len 32768 03/30 Read block 81 len 20480 04/30 Write block 396 len 32768 data: 16 13 11 15 16 11 10 14 05/30 Flush 06/30 Write block 365 len 40960 data: 16 19 17 17 22 20 17 19 19 20 07/30 Flush 08/30 Write block 137 len 16384 data: 15 15 16 15 09/30 Write block 590 len 32768 data: 19 20 18 20 19 15 14 10 10/30 Read block 381 len 36864 11/30 Read block 166 len 16384 12/30 Write block 155 len 8192 data: 13 11 13/30 Flush 14/30 Write block 575 len 24576 data: 17 19 12 8 7 10 15/30 Write block 37 len 20480 data: 19 12 13 12 12 16/30 Read block 166 len 32768 17/30 Read block 409 len 16384 18/30 Flush 19/30 Write block 474 len 24576 data: 9 7 9 8 9 9 20/30 Read block 429 len 24576 21/30 Read block 119 len 16384 22/30 Write block 498 len 24576 data: 5 7 9 8 10 11 23/30 Read block 112 len 40960 24/30 Read block 550 len 28672 25/30 Read block 457 len 32768 26/30 Read block 202 len 12288 27/30 Write block 431 len 36864 data: 11 12 13 17 17 14 14 14 10 28/30 Write block 62 len 20480 data: 19 16 13 16 17 29/30 Read block 230 len 24576 30/30 Write block 252 len 8192 data: 9 10 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 6 Done Done Done false 20 Acked 1019 Read 6 Done Done Done false 21 Acked 1020 Read 4 Done Done Done false 22 Acked 1021 Write 6 Done Done Done false 23 Acked 1022 Read 10 Done Done Done false 24 Acked 1023 Read 7 Done Done Done false 25 Acked 1024 Read 8 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Write 9 Done Done Done false 28 Acked 1027 Write 5 Done Done Done false 29 Acked 1028 Read 6 Done Done Done false 30 Acked 1029 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 11 11 11 33 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:01:33.443459117Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1956} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 192 192 192 383 383 383 F F F 2 040-059 192 192 192 383 383 383 F F F 3 060-079 188 188 188 376 376 376 T T T 4 080-099 187 187 187 370 370 370 F F F 5 100-119 187 187 187 370 370 370 F F F 6 120-139 192 192 192 382 382 382 F F F 7 140-159 192 192 192 382 382 382 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 189 189 189 377 377 377 F F F 10 200-219 188 188 188 374 374 374 F F F 11 220-239 188 188 188 375 375 375 F F F 12 240-259 188 188 188 371 371 371 T T T 13 260-279 190 190 190 378 378 378 F F F 14 280-299 190 190 190 378 378 378 F F F 15 300-319 189 189 189 377 377 377 F F F 16 320-339 186 186 186 369 369 369 F F F 17 340-359 186 186 186 369 369 369 F F F 18 360-379 192 192 192 381 381 381 F F F 19 380-399 192 192 192 380 380 380 F F F 20 400-419 192 192 192 380 380 380 F F F 21 420-439 190 190 190 378 378 378 T T T 22 440-459 190 190 190 378 378 378 F F F 23 460-479 191 191 191 379 379 379 T T T 24 480-499 186 186 186 369 369 369 T T T 25 500-519 188 188 188 373 373 373 T T T 26 520-539 190 190 190 378 378 378 F F F 27 540-559 192 192 192 380 380 380 F F F 28 560-579 192 192 192 383 383 383 F F F 29 580-599 192 192 192 383 383 383 F F F Max gen: 192, Max flush: 383 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 192 192 192 383 383 383 F F F 2 040-059 192 192 192 383 383 383 F F F 3 060-079 193 193 193 384 384 384 F F F 4 080-099 187 187 187 370 370 370 F F F 5 100-119 187 187 187 370 370 370 F F F 6 120-139 192 192 192 382 382 382 F F F 7 140-159 192 192 192 382 382 382 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 189 189 189 377 377 377 F F F 10 200-219 188 188 188 374 374 374 F F F 11 220-239 188 188 188 375 375 375 F F F 12 240-259 193 193 193 384 384 384 F F F 13 260-279 190 190 190 378 378 378 F F F 14 280-299 190 190 190 378 378 378 F F F 15 300-319 189 189 189 377 377 377 F F F 16 320-339 186 186 186 369 369 369 F F F 17 340-359 186 186 186 369 369 369 F F F 18 360-379 192 192 192 381 381 381 F F F 19 380-399 192 192 192 380 380 380 F F F 20 400-419 192 192 192 380 380 380 F F F 21 420-439 193 193 193 384 384 384 F F F 22 440-459 190 190 190 378 378 378 F F F 23 460-479 193 193 193 384 384 384 F F F 24 480-499 193 193 193 384 384 384 F F F 25 500-519 193 193 193 384 384 384 F F F 26 520-539 190 190 190 378 378 378 F F F 27 540-559 192 192 192 380 380 380 F F F 28 560-579 192 192 192 383 383 383 F F F 29 580-599 192 192 192 383 383 383 F F F Max gen: 193, Max flush: 384 Begin loop 97 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.701223084Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.701523244Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.701540822Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964} {"msg":"Crucible ad4849dd-6773-4a74-a040-cbe27b427699 has session id: 045dffbe-6602-4d39-8225-8b78b2d0e866","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.701982759Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Upstairs opts: Upstairs UUID: ad4849dd-6773-4a74-a040-cbe27b427699, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702028284Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Crucible stats registered with UUID: ad4849dd-6773-4a74-a040-cbe27b427699","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702051692Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702362211Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964} {"msg":"ad4849dd-6773-4a74-a040-cbe27b427699 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702384639Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702398127Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"0","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702415596Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"0","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702430034Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"1","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702444023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"1","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702457521Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.70247121Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702484729Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"io task","client":"1","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702498947Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"io task","client":"0","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.70257375Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"io task","client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ds_connection connected from Ok(127.0.0.1:46909)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702681439Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"io task","client":"1","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:01:47.702835634Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"io task","client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:01:47.70287778Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"io task","client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ds_connection connected from Ok(127.0.0.1:37020)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702906937Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"io task","client":"0","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:01:47.702937154Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702961551Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.702989099Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.703051613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"io task","client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.70348233Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"0","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.703509167Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"0","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.703525156Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"1","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.703539104Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"1","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.704096259Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"0","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.704208558Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"1","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.704528916Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"0","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.704552804Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"0","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.704568552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.704713068Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"1","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.704732346Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"1","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:47.704746854Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.704420742Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"io task","client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ds_connection connected from Ok(127.0.0.1:46818)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.704628161Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"io task","client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.705609554Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.705637681Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706166679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706663679Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706691007Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[0]R flush_numbers[0..12]: [370, 383, 383, 384, 370, 370, 382, 382, 357, 377, 374, 375]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706710335Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[0]R generation[0..12]: [187, 192, 192, 193, 187, 187, 192, 192, 180, 189, 188, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706728283Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706746241Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[1]R flush_numbers[0..12]: [370, 383, 383, 384, 370, 370, 382, 382, 357, 377, 374, 375]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706763809Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[1]R generation[0..12]: [187, 192, 192, 193, 187, 187, 192, 192, 180, 189, 188, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706781178Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706812655Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[2]R flush_numbers[0..12]: [370, 383, 383, 384, 370, 370, 382, 382, 357, 377, 374, 375]","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2024-07-09T22:01:57.706831553Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[2]R generation[0..12]: [187, 192, 192, 193, 187, 187, 192, 192, 180, 189, 188, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706852291Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706870209Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Max found gen is 194","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706890907Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706907875Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Next flush: 385","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706924464Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706941162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"":"downstairs","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.70695783Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706972989Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"0","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.706989757Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"1","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.707006336Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"client":"2","":"downstairs","session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.707023124Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.707038542Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"ad4849dd-6773-4a74-a040-cbe27b427699 is now active with session: 045dffbe-6602-4d39-8225-8b78b2d0e866","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.707054791Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.707069589Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964,"session_id":"045dffbe-6602-4d39-8225-8b78b2d0e866"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:01:57.707085188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 108 len 8192 data: 15 14 02/30 Write block 36 len 20480 data: 18 20 13 14 13 03/30 Write block 226 len 8192 data: 11 8 04/30 Read block 278 len 36864 05/30 Write block 373 len 36864 data: 20 21 19 17 12 10 13 12 12 06/30 Write block 527 len 24576 data: 18 18 16 13 16 14 07/30 Write block 350 len 4096 data: 14 08/30 Read block 421 len 20480 09/30 Write block 210 len 40960 data: 16 18 18 18 17 18 17 22 19 19 10/30 Flush 11/30 Write block 443 len 32768 data: 16 16 17 14 15 17 19 18 12/30 Write block 491 len 8192 data: 17 10 13/30 Read block 90 len 24576 14/30 Write block 253 len 40960 data: 11 9 13 13 12 11 11 10 9 11 15/30 Read block 88 len 24576 16/30 Read block 466 len 20480 17/30 Flush 18/30 Write block 400 len 24576 data: 17 12 11 15 13 12 19/30 Write block 591 len 24576 data: 21 19 21 20 16 15 20/30 Read block 280 len 40960 21/30 Flush 22/30 Write block 527 len 24576 data: 19 19 17 14 17 15 23/30 Write block 560 len 40960 data: 13 13 13 14 13 11 10 10 16 15 24/30 Read block 183 len 40960 25/30 Flush 26/30 Write block 150 len 16384 data: 12 14 12 11 27/30 Write block 541 len 8192 data: 13 12 28/30 Write block 213 len 20480 data: 19 18 19 18 23 29/30 Read block 120 len 28672 30/30 Read block 139 len 36864 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Write 4 Done Done Done false 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Write 5 Done Done Done false 29 Acked 1028 Read 7 Done Done Done false 30 Acked 1029 Read 9 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 0 1 Done 5 4 5 14 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:01:57.738314573Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1964} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 194 194 194 385 385 385 F F F 2 040-059 194 194 194 385 385 385 F F F 3 060-079 193 193 193 384 384 384 F F F 4 080-099 187 187 187 370 370 370 F F F 5 100-119 194 194 194 385 385 385 F F F 6 120-139 192 192 192 382 382 382 F F F 7 140-159 192 192 192 382 382 382 T T T 8 160-179 180 180 180 357 357 357 F F F 9 180-199 189 189 189 377 377 377 F F F 10 200-219 194 194 194 385 385 385 T T T 11 220-239 194 194 194 385 385 385 F F F 12 240-259 194 194 194 386 386 386 F F F 13 260-279 194 194 194 386 386 386 F F F 14 280-299 190 190 190 378 378 378 F F F 15 300-319 189 189 189 377 377 377 F F F 16 320-339 186 186 186 369 369 369 F F F 17 340-359 194 194 194 385 385 385 F F F 18 360-379 194 194 194 385 385 385 F F F 19 380-399 194 194 194 385 385 385 F F F 20 400-419 194 194 194 387 387 387 F F F 21 420-439 193 193 193 384 384 384 F F F 22 440-459 194 194 194 386 386 386 F F F 23 460-479 193 193 193 384 384 384 F F F 24 480-499 194 194 194 386 386 386 F F F 25 500-519 193 193 193 384 384 384 F F F 26 520-539 194 194 194 388 388 388 F F F 27 540-559 192 192 192 380 380 380 T T T 28 560-579 194 194 194 388 388 388 F F F 29 580-599 194 194 194 387 387 387 F F F Max gen: 194, Max flush: 388 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 194 194 194 385 385 385 F F F 2 040-059 194 194 194 385 385 385 F F F 3 060-079 193 193 193 384 384 384 F F F 4 080-099 187 187 187 370 370 370 F F F 5 100-119 194 194 194 385 385 385 F F F 6 120-139 192 192 192 382 382 382 F F F 7 140-159 195 195 195 389 389 389 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 189 189 189 377 377 377 F F F 10 200-219 195 195 195 389 389 389 F F F 11 220-239 194 194 194 385 385 385 F F F 12 240-259 194 194 194 386 386 386 F F F 13 260-279 194 194 194 386 386 386 F F F 14 280-299 190 190 190 378 378 378 F F F 15 300-319 189 189 189 377 377 377 F F F 16 320-339 186 186 186 369 369 369 F F F 17 340-359 194 194 194 385 385 385 F F F 18 360-379 194 194 194 385 385 385 F F F 19 380-399 194 194 194 385 385 385 F F F 20 400-419 194 194 194 387 387 387 F F F 21 420-439 193 193 193 384 384 384 F F F 22 440-459 194 194 194 386 386 386 F F F 23 460-479 193 193 193 384 384 384 F F F 24 480-499 194 194 194 386 386 386 F F F 25 500-519 193 193 193 384 384 384 F F F 26 520-539 194 194 194 388 388 388 F F F 27 540-559 195 195 195 389 389 389 F F F 28 560-579 194 194 194 388 388 388 F F F 29 580-599 194 194 194 387 387 387 F F F Max gen: 195, Max flush: 389 Begin loop 98 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.940134592Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.940409995Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.940426213Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972} {"msg":"Crucible 1243cd6a-5db5-46b2-9137-415c07c2845a has session id: 39e6e3c4-fb79-4e67-a241-c681f28c1470","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.940872179Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Upstairs opts: Upstairs UUID: 1243cd6a-5db5-46b2-9137-415c07c2845a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.940916805Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Crucible stats registered with UUID: 1243cd6a-5db5-46b2-9137-415c07c2845a","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.940939052Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941341212Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972} {"msg":"1243cd6a-5db5-46b2-9137-415c07c2845a active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.94136394Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941378139Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941396547Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941411745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"1","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941425594Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"1","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941438953Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"2","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941452501Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"2","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.94146575Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"io task","client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941479779Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"io task","client":"1","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941493628Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"io task","client":"2","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:02:11.941642783Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"io task","client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:02:11.941684518Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"io task","client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:02:11.941742863Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941764031Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941787808Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"ds_connection connected from Ok(127.0.0.1:60075)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941811356Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"io task","client":"1","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"ds_connection connected from Ok(127.0.0.1:33676)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.941953942Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"io task","client":"2","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.942011636Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"io task","client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.94258174Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"1","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.942620246Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"1","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.942649613Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"2","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.94267534Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"2","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.943167182Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"1","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.943247863Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"2","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.943678281Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"1","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.943704538Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"1","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.943719587Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.943743734Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"2","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.943758183Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"2","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:11.943771662Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.943568377Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"io task","client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"ds_connection connected from Ok(127.0.0.1:33582)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.943896745Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"io task","client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.944974008Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.944998555Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.945546581Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946088137Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946110185Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[0]R flush_numbers[0..12]: [370, 385, 385, 384, 370, 385, 382, 389, 357, 377, 389, 385]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946128353Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[0]R generation[0..12]: [187, 194, 194, 193, 187, 194, 192, 195, 180, 189, 195, 194]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946146111Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.94616398Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[1]R flush_numbers[0..12]: [370, 385, 385, 384, 370, 385, 382, 389, 357, 377, 389, 385]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946181538Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[1]R generation[0..12]: [187, 194, 194, 193, 187, 194, 192, 195, 180, 189, 195, 194]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946199016Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946216764Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[2]R flush_numbers[0..12]: [370, 385, 385, 384, 370, 385, 382, 389, 357, 377, 389, 385]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946234503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[2]R generation[0..12]: [187, 194, 194, 193, 187, 194, 192, 195, 180, 189, 195, 194]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946252011Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946276958Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946293737Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Generation requested: 196 >= found:196","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946310615Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Next flush: 390","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946327113Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946343482Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"":"downstairs","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.94636042Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946376339Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"0","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946391077Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"1","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946404486Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"client":"2","":"downstairs","session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946417924Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946430243Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"1243cd6a-5db5-46b2-9137-415c07c2845a is now active with session: 39e6e3c4-fb79-4e67-a241-c681f28c1470","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946443162Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.946455241Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972,"session_id":"39e6e3c4-fb79-4e67-a241-c681f28c1470"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:21.94646749Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 186 len 40960 data: 9 12 13 14 14 17 16 16 16 14 02/30 Write block 110 len 40960 data: 12 15 15 17 20 18 15 14 10 11 03/30 Read block 323 len 16384 04/30 Write block 545 len 8192 data: 13 17 05/30 Write block 231 len 12288 data: 12 11 12 06/30 Read block 368 len 20480 07/30 Write block 303 len 32768 data: 11 14 16 17 13 13 15 15 08/30 Write block 297 len 4096 data: 15 09/30 Flush 10/30 Write block 246 len 12288 data: 7 4 7 11/30 Read block 445 len 40960 12/30 Write block 184 len 24576 data: 8 9 10 13 14 15 13/30 Read block 170 len 4096 14/30 Write block 453 len 32768 data: 19 19 17 14 14 14 12 11 15/30 Write block 156 len 16384 data: 12 9 9 9 16/30 Write block 436 len 8192 data: 15 15 17/30 Flush 18/30 Write block 379 len 36864 data: 14 13 13 7 9 12 20 23 23 19/30 Read block 220 len 16384 20/30 Write block 581 len 12288 data: 13 15 14 21/30 Write block 199 len 20480 data: 16 15 11 13 16 22/30 Write block 339 len 8192 data: 14 13 23/30 Read block 461 len 12288 24/30 Write block 356 len 16384 data: 15 18 22 22 25/30 Read block 290 len 8192 26/30 Read block 59 len 8192 27/30 Read block 292 len 12288 28/30 Write block 186 len 12288 data: 11 14 15 29/30 Read block 467 len 36864 30/30 Read block 238 len 40960 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 9 Done Done Done false 19 Acked 1018 Read 4 Done Done Done false 20 Acked 1019 Write 3 Done Done Done false 21 Acked 1020 Write 5 Done Done Done false 22 Acked 1021 Write 2 Done Done Done false 23 Acked 1022 Read 3 Done Done Done false 24 Acked 1023 Write 4 Done Done Done false 25 Acked 1024 Read 2 Done Done Done false 26 Acked 1025 Read 2 Done Done Done false 27 Acked 1026 Read 3 Done Done Done false 28 Acked 1027 Write 3 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 13 12 12 37 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:02:21.973817539Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1972} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 194 194 194 385 385 385 F F F 2 040-059 194 194 194 385 385 385 F F F 3 060-079 193 193 193 384 384 384 F F F 4 080-099 187 187 187 370 370 370 F F F 5 100-119 196 196 196 390 390 390 F F F 6 120-139 192 192 192 382 382 382 F F F 7 140-159 196 196 196 391 391 391 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 196 196 196 391 391 391 T T T 10 200-219 195 195 195 389 389 389 T T T 11 220-239 196 196 196 390 390 390 F F F 12 240-259 196 196 196 391 391 391 F F F 13 260-279 194 194 194 386 386 386 F F F 14 280-299 196 196 196 390 390 390 F F F 15 300-319 196 196 196 390 390 390 F F F 16 320-339 186 186 186 369 369 369 T T T 17 340-359 194 194 194 385 385 385 T T T 18 360-379 194 194 194 385 385 385 T T T 19 380-399 194 194 194 385 385 385 T T T 20 400-419 194 194 194 387 387 387 F F F 21 420-439 196 196 196 391 391 391 F F F 22 440-459 196 196 196 391 391 391 F F F 23 460-479 196 196 196 391 391 391 F F F 24 480-499 194 194 194 386 386 386 F F F 25 500-519 193 193 193 384 384 384 F F F 26 520-539 194 194 194 388 388 388 F F F 27 540-559 196 196 196 390 390 390 F F F 28 560-579 194 194 194 388 388 388 F F F 29 580-599 194 194 194 387 387 387 T T T Max gen: 196, Max flush: 391 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 194 194 194 385 385 385 F F F 2 040-059 194 194 194 385 385 385 F F F 3 060-079 193 193 193 384 384 384 F F F 4 080-099 187 187 187 370 370 370 F F F 5 100-119 196 196 196 390 390 390 F F F 6 120-139 192 192 192 382 382 382 F F F 7 140-159 196 196 196 391 391 391 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 197 197 197 392 392 392 F F F 10 200-219 197 197 197 392 392 392 F F F 11 220-239 196 196 196 390 390 390 F F F 12 240-259 196 196 196 391 391 391 F F F 13 260-279 194 194 194 386 386 386 F F F 14 280-299 196 196 196 390 390 390 F F F 15 300-319 196 196 196 390 390 390 F F F 16 320-339 197 197 197 392 392 392 F F F 17 340-359 197 197 197 392 392 392 F F F 18 360-379 197 197 197 392 392 392 F F F 19 380-399 197 197 197 392 392 392 F F F 20 400-419 194 194 194 387 387 387 F F F 21 420-439 196 196 196 391 391 391 F F F 22 440-459 196 196 196 391 391 391 F F F 23 460-479 196 196 196 391 391 391 F F F 24 480-499 194 194 194 386 386 386 F F F 25 500-519 193 193 193 384 384 384 F F F 26 520-539 194 194 194 388 388 388 F F F 27 540-559 196 196 196 390 390 390 F F F 28 560-579 194 194 194 388 388 388 F F F 29 580-599 197 197 197 392 392 392 F F F Max gen: 197, Max flush: 392 Begin loop 99 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.235942393Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e5f834981d3a43d75fe6f427c4755541ac0c2270\",\n git_commit_timestamp: \"2024-07-09T21:07:43.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.23636922Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.236401277Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980} {"msg":"Crucible ce5ae035-292b-49bb-8838-59559bc9a40c has session id: 05e15895-ff4d-49f1-b21f-59a5cb749cde","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.236738154Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Upstairs opts: Upstairs UUID: ce5ae035-292b-49bb-8838-59559bc9a40c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.23678132Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Crucible stats registered with UUID: ce5ae035-292b-49bb-8838-59559bc9a40c","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.236804657Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237093139Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980} {"msg":"ce5ae035-292b-49bb-8838-59559bc9a40c active request set","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237167931Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237191199Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237216256Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237241944Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"1","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237267771Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"1","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237291289Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"2","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237315927Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"2","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237338834Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"io task","client":"2","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237363552Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"io task","client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237390199Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"io task","client":"1","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ds_connection connected from Ok(127.0.0.1:49812)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237433695Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"io task","client":"2","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ds_connection connected from Ok(127.0.0.1:46064)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237641225Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"io task","client":"1","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:02:36.237692189Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"io task","client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:02:36.237723026Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"io task","client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:02:36.237833525Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237851364Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.237865072Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.238035485Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"io task","client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.238422587Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"1","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.238444605Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"1","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.238459463Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"2","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.238473052Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"2","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.239092211Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"2","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.239116508Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"1","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.239623188Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"2","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.239648955Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"2","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.239664324Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.239677123Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"1","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.239691031Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"1","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:36.239708529Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.23904384Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"io task","client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ds_connection connected from Ok(127.0.0.1:57412)","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.239117883Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"io task","client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.240192707Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.240227503Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.240931953Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.2414653Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241490728Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[0]R flush_numbers[0..12]: [370, 385, 385, 384, 370, 390, 382, 391, 357, 392, 392, 390]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241507106Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[0]R generation[0..12]: [187, 194, 194, 193, 187, 196, 192, 196, 180, 197, 197, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241521685Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241536253Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[1]R flush_numbers[0..12]: [370, 385, 385, 384, 370, 390, 382, 391, 357, 392, 392, 390]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241550292Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[1]R generation[0..12]: [187, 194, 194, 193, 187, 196, 192, 196, 180, 197, 197, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241564251Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241578769Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[2]R flush_numbers[0..12]: [370, 385, 385, 384, 370, 390, 382, 391, 357, 392, 392, 390]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241592838Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[2]R generation[0..12]: [187, 194, 194, 193, 187, 196, 192, 196, 180, 197, 197, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241610556Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-09T22:02:46.241625305Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241642023Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Generation requested: 198 >= found:198","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241655522Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Next flush: 393","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.24166874Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241681959Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"":"downstairs","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241695207Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241707216Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"0","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241720565Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"1","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241733914Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"client":"2","":"downstairs","session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241747762Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241760061Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"ce5ae035-292b-49bb-8838-59559bc9a40c is now active with session: 05e15895-ff4d-49f1-b21f-59a5cb749cde","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.24177292Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241785169Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980,"session_id":"05e15895-ff4d-49f1-b21f-59a5cb749cde"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2024-07-09T22:02:46.241797447Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 344 len 24576 data: 15 15 16 12 12 13 02/30 Write block 48 len 4096 data: 13 03/30 Read block 121 len 32768 04/30 Write block 299 len 32768 data: 16 17 14 11 12 15 17 18 05/30 Flush 06/30 Write block 104 len 8192 data: 14 13 07/30 Write block 87 len 32768 data: 17 16 13 10 9 11 11 11 08/30 Flush 09/30 Write block 357 len 32768 data: 19 23 23 20 15 16 14 14 10/30 Write block 266 len 12288 data: 14 13 10 11/30 Read block 79 len 36864 12/30 Read block 567 len 20480 13/30 Read block 586 len 32768 14/30 Write block 595 len 8192 data: 17 16 15/30 Read block 151 len 20480 16/30 Flush 17/30 Write block 120 len 20480 data: 10 10 12 16 17 18/30 Read block 584 len 40960 19/30 Read block 428 len 12288 20/30 Write block 506 len 24576 data: 15 13 16 14 15 15 21/30 Write block 374 len 40960 data: 22 20 18 13 11 15 14 14 8 10 22/30 Write block 118 len 16384 data: 11 12 11 11 23/30 Read block 263 len 4096 24/30 Write block 121 len 12288 data: 12 13 17 25/30 Write block 523 len 4096 data: 11 26/30 Flush 27/30 Write block 33 len 8192 data: 15 16 28/30 Read block 468 len 8192 29/30 Read block 429 len 32768 30/30 Read block 418 len 36864 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Read 8 Done Done Done false 30 Acked 1029 Read 9 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 1 1 Done 4 4 3 11 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-09T22:02:46.272755579Z","hostname":"w-01J2CNYPQGXQVK3WVQV8ZVEK6G","pid":1980} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 194 194 194 385 385 385 T T T 2 040-059 198 198 198 393 393 393 F F F 3 060-079 193 193 193 384 384 384 F F F 4 080-099 198 198 198 394 394 394 F F F 5 100-119 198 198 198 396 396 396 F F F 6 120-139 198 198 198 396 396 396 F F F 7 140-159 196 196 196 391 391 391 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 197 197 197 392 392 392 F F F 10 200-219 197 197 197 392 392 392 F F F 11 220-239 196 196 196 390 390 390 F F F 12 240-259 196 196 196 391 391 391 F F F 13 260-279 198 198 198 395 395 395 F F F 14 280-299 198 198 198 393 393 393 F F F 15 300-319 198 198 198 393 393 393 F F F 16 320-339 197 197 197 392 392 392 F F F 17 340-359 198 198 198 395 395 395 F F F 18 360-379 198 198 198 396 396 396 F F F 19 380-399 198 198 198 396 396 396 F F F 20 400-419 194 194 194 387 387 387 F F F 21 420-439 196 196 196 391 391 391 F F F 22 440-459 196 196 196 391 391 391 F F F 23 460-479 196 196 196 391 391 391 F F F 24 480-499 194 194 194 386 386 386 F F F 25 500-519 198 198 198 396 396 396 F F F 26 520-539 198 198 198 396 396 396 F F F 27 540-559 196 196 196 390 390 390 F F F 28 560-579 194 194 194 388 388 388 F F F 29 580-599 198 198 198 395 395 395 F F F Max gen: 198, Max flush: 396 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 370 370 370 F F F 1 020-039 199 199 199 397 397 397 F F F 2 040-059 198 198 198 393 393 393 F F F 3 060-079 193 193 193 384 384 384 F F F 4 080-099 198 198 198 394 394 394 F F F 5 100-119 198 198 198 396 396 396 F F F 6 120-139 198 198 198 396 396 396 F F F 7 140-159 196 196 196 391 391 391 F F F 8 160-179 180 180 180 357 357 357 F F F 9 180-199 197 197 197 392 392 392 F F F 10 200-219 197 197 197 392 392 392 F F F 11 220-239 196 196 196 390 390 390 F F F 12 240-259 196 196 196 391 391 391 F F F 13 260-279 198 198 198 395 395 395 F F F 14 280-299 198 198 198 393 393 393 F F F 15 300-319 198 198 198 393 393 393 F F F 16 320-339 197 197 197 392 392 392 F F F 17 340-359 198 198 198 395 395 395 F F F 18 360-379 198 198 198 396 396 396 F F F 19 380-399 198 198 198 396 396 396 F F F 20 400-419 194 194 194 387 387 387 F F F 21 420-439 196 196 196 391 391 391 F F F 22 440-459 196 196 196 391 391 391 F F F 23 460-479 196 196 196 391 391 391 F F F 24 480-499 194 194 194 386 386 386 F F F 25 500-519 198 198 198 396 396 396 F F F 26 520-539 198 198 198 396 396 396 F F F 27 540-559 196 196 196 390 390 390 F F F 28 560-579 194 194 194 388 388 388 F F F 29 580-599 198 198 198 395 395 395 F F F Max gen: 199, Max flush: 397 40:05 Test duration Test completed real 40:05.192874194 user 41.456955996 sys 18.812479834 trap 0.057013316 tflt 0.126169026 dflt 0.136422330 kflt 0.007295223 lock 63:25:39.955623815 slp 3:17:14.833420702 lat 8.028984037 stop 0.065737699