Turn off color for downstairs dump Removing existing region "/var/tmp/test_repair" Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/test_live_repair/test_repair_dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.27313623 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.020679258 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.02054026 Created 3 regions /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1201 1202 1203 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.170874384Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.235924145Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.237747937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.23782401Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.237842078Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.237864306Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206} {"msg":"Crucible 682a8fdc-1faa-464c-99e6-abce712f5e3a has session id: c5cf854e-fef4-4470-bd50-a248a9358165","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.238755589Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Upstairs opts: Upstairs UUID: 682a8fdc-1faa-464c-99e6-abce712f5e3a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.238817183Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Crucible stats registered with UUID: 682a8fdc-1faa-464c-99e6-abce712f5e3a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.238843101Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:29.238863659Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.239680148Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.23977661Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.239813757Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.239962953Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.240023998Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206} {"msg":"682a8fdc-1faa-464c-99e6-abce712f5e3a active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.240094952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.24011605Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"client":"0","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.240133978Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"client":"1","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.240151447Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"client":"2","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.240168515Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.240186034Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.240203143Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"ds_connection connected from Ok(127.0.0.1:33269)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.240405395Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"ds_connection connected from Ok(127.0.0.1:46613)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.240447861Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"ds_connection connected from Ok(127.0.0.1:64788)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.240571071Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.243020687Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"client":"1","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.243054594Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"client":"0","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.243078502Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"client":"2","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244300415Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244330933Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244442483Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244466251Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244486489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244504898Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244523036Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244542744Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244561033Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244578911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.24459846Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244614958Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244631817Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244647845Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244667574Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"":"downstairs","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244685192Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244700951Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"client":"0","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244718179Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"client":"1","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244734948Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"client":"2","":"downstairs","session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244757346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg"Wait for a query_work_queue command to finish before sending IO :"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244788313Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"682a8fdc-1faa-464c-99e6-abce712f5e3a is now active with session: c5cf854e-fef4-4470-bd50-a248a9358165","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244808971Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.2448271Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244849598Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206,"session_id":"c5cf854e-fef4-4470-bd50-a248a9358165"} {"msg":"Activated sub_volume 682a8fdc-1faa-464c-99e6-abce712f5e3a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:29.244872506Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1206} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Fill test Read and Verify all blocks (0..600 range:false) Wrote out file "/tmp/test_live_repair/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Read 88 Done Done Done false 1005 Acked Read 256 Done Done Done false 1006 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1006 1005 1004 1003 1002 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1007 1006 1005 1004 1003 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program Begin loop 1 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.357233311Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:33.401138635Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.401561108Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.432623701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.433309401Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.433366726Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.433386215Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.433413352Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208} {"msg":"Crucible 42cd4344-d838-49eb-ac9e-c8503bd4d7ae has session id: 2c4bc65b-6524-4162-ad28-e91643d383eb","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.433891251Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Upstairs opts: Upstairs UUID: 42cd4344-d838-49eb-ac9e-c8503bd4d7ae, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.433963454Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Crucible stats registered with UUID: 42cd4344-d838-49eb-ac9e-c8503bd4d7ae","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.433996571Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:33.434024729Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434231741Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434269578Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434294435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434495808Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"task":"crutest"} {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434524225Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208} {"msg":"42cd4344-d838-49eb-ac9e-c8503bd4d7ae active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434593589Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434614937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"client":"0","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434632466Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"client":"1","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434649445Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"client":"2","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434666103Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434683491Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.43470166Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:50783)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434936989Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:45986)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434978686Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:60636)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.434999104Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436101568Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"client":"1","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436135825Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"client":"2","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.43619907Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"client":"0","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436713934Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436752771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436772889Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436791538Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436811476Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436829555Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436847543Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436867111Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.43688517Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436913937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436949174Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {Wait for a query_work_queue command to finish before sending IO "msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.436980691Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437013499Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437045906Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437081973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"":"downstairs","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437117489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437148307Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"client":"0","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437180784Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 client":"1","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437218481Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"client":"2","":"downstairs","session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437239239Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437255687Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"42cd4344-d838-49eb-ac9e-c8503bd4d7ae is now active with session: 2c4bc65b-6524-4162-ad28-e91643d383eb","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437273716Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437289104Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208,"session_id":"2c4bc65b-6524-4162-ad28-e91643d383eb"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437305203Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208} {"msg":"Activated sub_volume 42cd4344-d838-49eb-ac9e-c8503bd4d7ae","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:33.437319432Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 408 len 36864 data: 2 2 2 2 2 2 2 2 2 02/30 Write block 509 len 4096 data: 2 03/30 Read block 510 len 24576 04/30 Read block 379 len 24576 05/30 Write block 330 len 12288 data: 2 2 2 06/30 Write block 461 len 36864 data: 2 2 2 2 2 2 2 2 2 07/30 Write block 83 len 20480 data: 2 2 2 2 2 08/30 Read block 396 len 32768 09/30 Write block 272 len 24576 data: 2 2 2 2 2 2 10/30 Read block 159 len 28672 11/30 Read block 363 len 16384 12/30 Read block 259 len 36864 13/30 Write block 15 len 28672 data: 2 2 2 2 2 2 2 14/30 Write block 570 len 8192 data: 2 2 15/30 Read block 394 len 8192 16/30 Read block 219 len 40960 17/30 Read block 589 len 40960 18/30 Write block 394 len 16384 data: 2 2 2 2 19/30 Write block 352 len 24576 data: 2 2 2 2 2 2 20/30 Read block 158 len 40960 21/30 Read block 537 len 4096 22/30 Flush 23/30 Write block 402 len 8192 data: 2 2 24/30 Read block 548 len 4096 25/30 Flush 26/30 Write block 346 len 16384 data: 2 2 2 2 27/30 Write block 367 len 12288 data: 2 2 2 28/30 Read block 528 len 24576 29/30 Write block 216 len 40960 data: 2 2 2 2 2 2 2 2 2 2 30/30 Write block 350 len 40960 data: 2 2 3 3 3 3 3 3 2 2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Write 3 Done Done Done false 1005 Acked Write 9 Done Done Done false 1006 Acked Write 5 Done Done Done false 1007 Acked Read 8 Done Done Done false 1008 Acked Write 6 Done Done Done false 1009 Acked Read 7 Done Done Done false 1010 Acked Read 4 Done Done Done false 1011 Acked Read 9 Done Done Done false 1012 Acked Write 7 Done Done Done false 1013 Acked Write 2 Done Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Read 10 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 16 2 2 20 Done 14 28 28 70 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:33.456110714Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1208} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 2 2 1 2 2 T F F <--- 1 020-039 1 2 2 1 2 2 T F F <--- 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 1 2 2 1 2 2 T F F <--- 5 100-119 1 1 1 1 1 1 F F F 6 120-139 1 1 1 1 1 1 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F T T <--- 11 220-239 1 1 1 1 1 1 F T T <--- 12 240-259 1 1 1 1 1 1 F F F 13 260-279 1 2 2 1 2 2 T F F <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 1 2 2 1 2 2 T F F <--- 17 340-359 1 2 2 1 2 2 F T T <--- 18 360-379 1 1 1 1 1 1 F T T <--- 19 380-399 1 2 2 1 2 2 F F F <--- 20 400-419 1 2 2 1 3 3 T F F <--- 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 2 2 1 2 2 T F F <--- 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 2 2 1 2 2 T F F <--- 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 2 2 1 2 2 T F F <--- 29 580-599 1 1 1 1 1 1 F F F Max gen: 2, Max flush: 3 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 2 2 2 2 2 2 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 2 2 2 2 2 2 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 1 1 1 1 1 1 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 3 3 3 4 4 4 F F F 11 220-239 3 3 3 4 4 4 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 2 2 2 2 2 2 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 2 2 2 2 2 2 F F F 17 340-359 3 3 3 4 4 4 F F F 18 360-379 3 3 3 4 4 4 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 2 2 2 3 3 3 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 2 2 2 2 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 2 2 2 2 2 2 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 2 2 2 2 2 2 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 3, Max flush: 4 Begin loop 2 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.063116493Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.108392688Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.109117245Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.109164551Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.109181069Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.109202827Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216} {"msg":"Crucible b2d6863c-04f7-4e2d-b29a-f714066543f8 has session id: d22f9229-607c-4029-9479-b0a8afec1c19","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.109808404Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Upstairs opts: Upstairs UUID: b2d6863c-04f7-4e2d-b29a-f714066543f8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.109851491Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Crucible stats registered with UUID: b2d6863c-04f7-4e2d-b29a-f714066543f8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.109870679Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:38.109887088Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110131426Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"io task","client":"0","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110181512Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"io task","client":"1","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.11020873Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"io task","client":"2","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110353477Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"task":"crutest"} {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110404153Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216} {"msg":"b2d6863c-04f7-4e2d-b29a-f714066543f8 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110445419Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110465107Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"client":"0","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110483246Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"client":"1","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110509014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"client":"2","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110540581Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"io task","client":"2","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110573098Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"io task","client":"0","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110610695Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"io task","client":"1","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"ds_connection connected from Ok(127.0.0.1:33102)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110646991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"io task","client":"0","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"ds_connection connected from Ok(127.0.0.1:49310)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110683708Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"io task","client":"2","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"ds_connection connected from Ok(127.0.0.1:49292)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.110721425Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"io task","client":"1","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.111983175Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"client":"1","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112125682Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"client":"0","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112167749Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"client":"2","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112537847Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112688924Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"[0]R flush_numbers[0..12]: [2, 2, 1, 1, 2, 1, 1, 1, 1, 1, 4, 4]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112804394Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"[0]R generation[0..12]: [2, 2, 1, 1, 2, 1, 1, 1, 1, 1, 3, 3]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112828661Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.11284919Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"[1]R flush_numbers[0..12]: [2, 2, 1, 1, 2, 1, 1, 1, 1, 1, 4, 4]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112867608Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"[1]R generation[0..12]: [2, 2, 1, 1, 2, 1, 1, 1, 1, 1, 3, 3]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112885776Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112910014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"[2]R flush_numbers[0..12]: [2, 2, 1, 1, 2, 1, 1, 1, 1, 1, 4, 4]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112928433Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"[2]R generation[0..12]: [2, 2, 1, 1, 2, 1, 1, 1, 1, 1, 3, 3]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112946161Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.112968549Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:15:38.112997467Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113025824Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Next flush: 5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113053562Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113082439Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"":"downstairs","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113112576Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113139924Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"client":"0","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113167862Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"client":"1","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.11318617Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"client":"2","":"downstairs","session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113203489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113219607Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"b2d6863c-04f7-4e2d-b29a-f714066543f8 is now active with session: d22f9229-607c-4029-9479-b0a8afec1c19","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113237456Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113256364Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216,"session_id":"d22f9229-607c-4029-9479-b0a8afec1c19"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113272983Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216} {"msg":"Activated sub_volume b2d6863c-04f7-4e2d-b29a-f714066543f8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:38.113287211Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 405 len 16384 data: 2 2 2 3 02/30 Read block 222 len 20480 03/30 Read block 295 len 16384 04/30 Read block 191 len 32768 05/30 Write block 165 len 8192 data: 2 2 06/30 Read block 579 len 12288 07/30 Read block 544 len 4096 08/30 Read block 411 len 4096 09/30 Write block 135 len 8192 data: 2 2 10/30 Read block 555 len 16384 11/30 Read block 241 len 28672 12/30 Write block 173 len 40960 data: 2 2 2 2 2 2 2 2 2 2 13/30 Read block 305 len 36864 14/30 Read block 198 len 20480 15/30 Write block 587 len 12288 data: 2 2 2 16/30 Write block 74 len 20480 data: 2 2 2 2 2 17/30 Read block 211 len 16384 18/30 Read block 229 len 8192 19/30 Read block 68 len 36864 20/30 Read block 497 len 12288 21/30 Write block 406 len 24576 data: 3 3 4 3 3 3 22/30 Read block 329 len 24576 23/30 Read block 515 len 32768 24/30 Read block 559 len 40960 25/30 Write block 185 len 28672 data: 2 2 2 2 2 2 2 26/30 Write block 360 len 28672 data: 2 2 2 2 2 2 2 27/30 Read block 197 len 32768 28/30 Read block 48 len 8192 29/30 Write block 176 len 24576 data: 3 3 3 3 3 3 30/30 Write block 212 len 16384 data: 2 2 2 2 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 5 Done Done Sent false 1002 Acked Read 4 Done Done Sent false 1003 Acked Read 8 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Read 3 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Write 2 Done Done Sent false 1009 Acked Read 4 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Read 4 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 29 33 Done 28 28 1 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:38.132384167Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1216} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 2 2 2 2 2 2 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 T T F <--- 4 080-099 2 2 2 2 2 2 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 1 1 1 1 1 1 T T F <--- 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 T T F <--- 9 180-199 1 1 1 1 1 1 T T F <--- 10 200-219 3 3 3 4 4 4 T T F <--- 11 220-239 3 3 3 4 4 4 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 2 2 2 2 2 2 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 2 2 2 2 2 2 F F F 17 340-359 3 3 3 4 4 4 F F F 18 360-379 3 3 3 4 4 4 T T F <--- 19 380-399 2 2 2 2 2 2 F F F 20 400-419 2 2 2 3 3 3 T T T 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 2 2 2 2 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 2 2 2 2 2 2 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 2 2 2 2 2 2 F F F 29 580-599 1 1 1 1 1 1 T T F <--- Max gen: 3, Max flush: 4 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 2 2 2 2 2 2 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 4 4 4 5 5 5 F F F 4 080-099 2 2 2 2 2 2 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 4 4 4 5 5 5 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 4 4 4 5 5 5 F F F 9 180-199 4 4 4 5 5 5 F F F 10 200-219 4 4 4 5 5 5 F F F 11 220-239 3 3 3 4 4 4 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 2 2 2 2 2 2 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 2 2 2 2 2 2 F F F 17 340-359 3 3 3 4 4 4 F F F 18 360-379 4 4 4 5 5 5 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 4 4 4 5 5 5 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 2 2 2 2 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 2 2 2 2 2 2 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 2 2 2 2 2 2 F F F 29 580-599 4 4 4 5 5 5 F F F Max gen: 4, Max flush: 5 Begin loop 3 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.516148292Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.560972186Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.56172635Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.561782795Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.561803173Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.561830561Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224} {"msg":"Crucible bc268a48-d313-494a-8f18-4a61096a4519 has session id: f287e5a5-990e-477b-be22-666d9431da89","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.562377663Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Upstairs opts: Upstairs UUID: bc268a48-d313-494a-8f18-4a61096a4519, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.562432748Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Crucible stats registered with UUID: bc268a48-d313-494a-8f18-4a61096a4519","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.562462266Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:42.562487534Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.562694546Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"io task","client":"0","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.562736412Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"io task","client":"1","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.562769669Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"io task","client":"2","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.562906957Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"task":"crutest"} {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.562928085Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224} {"msg":"bc268a48-d313-494a-8f18-4a61096a4519 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.56299589Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.563027007Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"client":"0","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.563047045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"client":"1","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.563064324Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"client":"2","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.563081312Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"io task","client":"2","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.563135197Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"io task","client":"0","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.563168414Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"io task","client":"1","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"ds_connection connected from Ok(127.0.0.1:42901)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.563201532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"io task","client":"2","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"ds_connection connected from Ok(127.0.0.1:47320)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.563235778Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"io task","client":"0","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"ds_connection connected from Ok(127.0.0.1:38790)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.563271306Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"io task","client":"1","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.564423075Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"client":"0","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.564458622Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"client":"2","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.565085027Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.565115435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.565595213Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"client":"1","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"[0]R flush_numbers[0..12]: [2, 2, 1, 5, 2, 1, 5, 1, 5, 5, 5, 4]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.56609052Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"[0]R generation[0..12]: [2, 2, 1, 4, 2, 1, 4, 1, 4, 4, 4, 3]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566124907Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566150484Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"[1]R flush_numbers[0..12]: [2, 2, 1, 5, 2, 1, 5, 1, 5, 5, 5, 4]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566173153Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"[1]R generation[0..12]: [2, 2, 1, 4, 2, 1, 4, 1, 4, 4, 4, 3]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566195751Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566219889Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"[2]R flush_numbers[0..12]: [2, 2, 1, 5, 2, 1, 5, 1, 5, 5, 5, 4]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566242256Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"[2]R generation[0..12]: [2, 2, 1, 4, 2, 1, 4, 1, 4, 4, 4, 3]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566264674Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566295512Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Max found gen is 5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Generation requested: 6 >= found:5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566337088Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Next flush: 6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566356946Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566377635Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"":"downstairs","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566398973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566421751Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"client":"0","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566443059Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"client":"1","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566463767Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"client":"2","":"downstairs","session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :30,"time":"2025-12-24T00:15:42.566489165Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566509413Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"bc268a48-d313-494a-8f18-4a61096a4519 is now active with session: f287e5a5-990e-477b-be22-666d9431da89","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566531931Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.56655119Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224,"session_id":"f287e5a5-990e-477b-be22-666d9431da89"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566571108Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224} {"msg":"Activated sub_volume bc268a48-d313-494a-8f18-4a61096a4519","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:42.566588626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 196 len 16384 data: 2 2 2 2 02/30 Write block 359 len 12288 data: 3 3 3 03/30 Write block 198 len 16384 data: 3 3 2 2 04/30 Read block 237 len 28672 05/30 Read block 33 len 8192 06/30 Write block 534 len 12288 data: 2 2 2 07/30 Read block 149 len 28672 08/30 Read block 108 len 28672 09/30 Write block 257 len 28672 data: 2 2 2 2 2 2 2 10/30 Read block 29 len 32768 11/30 Flush 12/30 Flush 13/30 Write block 482 len 40960 data: 2 2 2 2 2 2 2 2 2 2 14/30 Read block 167 len 36864 15/30 Write block 68 len 32768 data: 2 2 2 2 2 2 3 3 16/30 Write block 161 len 20480 data: 2 2 2 2 3 17/30 Read block 331 len 4096 18/30 Read block 549 len 12288 19/30 Read block 25 len 16384 20/30 Write block 505 len 8192 data: 2 2 21/30 Read block 556 len 32768 22/30 Read block 210 len 32768 23/30 Write block 182 len 8192 data: 3 2 24/30 Read block 148 len 16384 25/30 Write block 403 len 36864 data: 3 2 3 4 4 5 4 4 4 26/30 Read block 504 len 40960 27/30 Read block 85 len 8192 28/30 Write block 493 len 36864 data: 2 2 2 2 2 2 2 2 2 29/30 Read block 109 len 12288 30/30 Write block 100 len 36864 data: 2 2 2 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 3 Done Sent Done false 1002 Acked Write 4 Done Sent Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Read 2 Done Sent Done false 1005 Acked Write 3 Done Sent Done false 1006 Acked Read 7 Done Sent Done false 1007 Acked Read 7 Done Sent Done false 1008 Acked Write 7 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Write 9 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1011)) None Some(JobId(1011)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:42.586158061Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1224} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 2 2 2 2 2 2 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 4 4 4 5 5 5 T F T <--- 4 080-099 2 2 2 2 2 2 F F F 5 100-119 1 1 1 1 1 1 T F T <--- 6 120-139 4 4 4 5 5 5 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 4 4 4 5 5 5 T F T <--- 9 180-199 6 4 6 6 5 6 T T T <--- 10 200-219 6 4 6 6 5 6 F F F <--- 11 220-239 3 3 3 4 4 4 F F F 12 240-259 6 1 6 6 1 6 F F F <--- 13 260-279 6 2 6 6 2 6 F F F <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 2 2 2 2 2 2 F F F 17 340-359 6 3 6 6 4 6 F F F <--- 18 360-379 6 4 6 6 5 6 F F F <--- 19 380-399 2 2 2 2 2 2 F F F 20 400-419 4 4 4 5 5 5 T F T <--- 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 2 2 2 2 F F F 24 480-499 1 1 1 1 1 1 T F T <--- 25 500-519 2 2 2 2 2 2 T F T <--- 26 520-539 6 1 6 6 1 6 F F F <--- 27 540-559 1 1 1 1 1 1 F F F 28 560-579 2 2 2 2 2 2 F F F 29 580-599 4 4 4 5 5 5 F F F Max gen: 6, Max flush: 6 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 2 2 2 2 2 2 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 7 7 7 7 7 7 F F F 4 080-099 2 2 2 2 2 2 F F F 5 100-119 7 7 7 7 7 7 F F F 6 120-139 4 4 4 5 5 5 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 7 7 7 7 7 7 F F F 9 180-199 7 7 7 7 7 7 F F F 10 200-219 6 6 6 6 6 6 F F F 11 220-239 3 3 3 4 4 4 F F F 12 240-259 6 6 6 6 6 6 F F F 13 260-279 6 6 6 6 6 6 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 2 2 2 2 2 2 F F F 17 340-359 6 6 6 6 6 6 F F F 18 360-379 6 6 6 6 6 6 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 7 7 7 7 7 7 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 2 2 2 2 F F F 24 480-499 7 7 7 7 7 7 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 6 6 6 6 6 6 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 2 2 2 2 2 2 F F F 29 580-599 4 4 4 5 5 5 F F F Max gen: 7, Max flush: 7 Begin loop 4 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:46.989490489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.035049799Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.035787925Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.035835351Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.035851809Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.035873867Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232} {"msg":"Crucible 9295288b-625e-47ce-932b-073988d016f4 has session id: b012b5be-9c87-425e-9ac8-76b4a6554162","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.036448367Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Upstairs opts: Upstairs UUID: 9295288b-625e-47ce-932b-073988d016f4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.036512892Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Crucible stats registered with UUID: 9295288b-625e-47ce-932b-073988d016f4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.036536799Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:47.036553598Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.036709234Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"io task","client":"0","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.03675892Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"io task","client":"1","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.036793437Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.036980371Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"task":"crutest"} {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.037059284Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232} {"msg":"9295288b-625e-47ce-932b-073988d016f4 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.037090191Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.037120448Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"client":"0","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.037153176Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"client":"1","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.037188353Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"client":"2","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.03722065Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.037276635Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"io task","client":"0","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.037313282Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"io task","client":"1","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"ds_connection connected from Ok(127.0.0.1:58058)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.037352028Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"ds_connection connected from Ok(127.0.0.1:62630)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.037474408Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"io task","client":"0","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"ds_connection connected from Ok(127.0.0.1:49910)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.037520224Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"io task","client":"1","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.038444783Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"client":"0","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.038564323Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"client":"1","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039013424Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039088207Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"client":"2","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039190248Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"[0]R flush_numbers[0..12]: [2, 2, 1, 7, 2, 7, 5, 1, 7, 7, 6, 4]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039584184Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"[0]R generation[0..12]: [2, 2, 1, 7, 2, 7, 4, 1, 7, 7, 6, 3]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039612482Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.03963323Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"[1]R flush_numbers[0..12]: [2, 2, 1, 7, 2, 7, 5, 1, 7, 7, 6, 4]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039652298Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"[1]R generation[0..12]: [2, 2, 1, 7, 2, 7, 4, 1, 7, 7, 6, 3]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039670366Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039689915Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"[2]R flush_numbers[0..12]: [2, 2, 1, 7, 2, 7, 5, 1, 7, 7, 6, 4]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039708033Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"[2]R generation[0..12]: [2, 2, 1, 7, 2, 7, 4, 1, 7, 7, 6, 3]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039730581Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039762228Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Max found gen is 8","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2025-12-24T00:15:47.039792346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039826383Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Next flush: 8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.03985825Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039890717Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"":"downstairs","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039924054Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.039954941Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"client":"0","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2025-12-24T00:15:47.039987869Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"client":"1","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.040043184Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"client":"2","":"downstairs","session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.040075251Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.04009282Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"9295288b-625e-47ce-932b-073988d016f4 is now active with session: b012b5be-9c87-425e-9ac8-76b4a6554162","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.040110988Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.040126617Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232,"session_id":"b012b5be-9c87-425e-9ac8-76b4a6554162"} {"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.040142825Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232} {"msg":"Activated sub_volume 9295288b-625e-47ce-932b-073988d016f4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:47.040159814Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 212 len 20480 data: 3 3 3 3 3 02/30 Write block 335 len 12288 data: 2 2 2 03/30 Read block 490 len 28672 04/30 Read block 528 len 24576 05/30 Write block 40 len 24576 data: 2 2 2 2 2 2 06/30 Write block 145 len 4096 data: 2 07/30 Read block 465 len 8192 08/30 Write block 224 len 20480 data: 3 3 2 2 2 09/30 Read block 392 len 16384 10/30 Write block 133 len 12288 data: 2 2 3 11/30 Flush 12/30 Write block 269 len 36864 data: 2 2 2 3 3 3 3 3 3 13/30 Write block 29 len 20480 data: 2 2 2 2 2 14/30 Write block 352 len 28672 data: 4 4 4 4 4 4 3 15/30 Write block 98 len 16384 data: 2 2 3 3 16/30 Write block 58 len 12288 data: 2 2 2 17/30 Write block 413 len 16384 data: 3 3 3 3 18/30 Read block 298 len 4096 19/30 Write block 277 len 8192 data: 4 2 20/30 Write block 586 len 8192 data: 2 3 21/30 Read block 578 len 24576 22/30 Read block 342 len 20480 23/30 Write block 548 len 20480 data: 2 2 2 2 2 24/30 Write block 334 len 20480 data: 2 3 3 3 2 25/30 Flush 26/30 Write block 246 len 16384 data: 2 2 2 2 27/30 Write block 42 len 20480 data: 3 3 3 3 2 28/30 Read block 57 len 20480 29/30 Read block 156 len 12288 30/30 Read block 289 len 4096 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Sent false 1001 Acked Write 3 Done Done Sent false 1002 Acked Read 7 Done Done Sent false 1003 Acked Read 6 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Read 2 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Read 4 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Write 3 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Read 5 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 30 31 Done 30 29 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:47.056312086Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1232} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 8 8 2 9 9 2 F F F <--- 2 040-059 8 8 1 9 9 1 T T F <--- 3 060-079 8 8 7 9 9 7 F F F <--- 4 080-099 8 8 2 9 9 2 F F F <--- 5 100-119 8 8 7 9 9 7 F F F <--- 6 120-139 8 8 4 8 8 5 F F F <--- 7 140-159 8 8 1 8 8 1 F F F <--- 8 160-179 7 7 7 7 7 7 F F F 9 180-199 7 7 7 7 7 7 F F F 10 200-219 8 8 6 8 8 6 F F F <--- 11 220-239 8 8 3 8 8 4 F F F <--- 12 240-259 6 6 6 6 6 6 T T F <--- 13 260-279 8 8 6 9 9 6 F F F <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 8 8 2 9 9 2 F F F <--- 17 340-359 8 8 6 9 9 6 F F F <--- 18 360-379 6 6 6 6 6 6 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 8 8 7 9 9 7 F F F <--- 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 2 2 2 2 F F F 24 480-499 7 7 7 7 7 7 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 6 6 6 6 6 6 F F F 27 540-559 8 8 1 9 9 1 F F F <--- 28 560-579 2 2 2 2 2 2 F F F 29 580-599 8 8 4 9 9 5 F F F <--- Max gen: 8, Max flush: 9 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 8 8 8 9 9 9 F F F 2 040-059 9 9 9 10 10 10 F F F 3 060-079 8 8 8 9 9 9 F F F 4 080-099 8 8 8 9 9 9 F F F 5 100-119 8 8 8 9 9 9 F F F 6 120-139 8 8 8 8 8 8 F F F 7 140-159 8 8 8 8 8 8 F F F 8 160-179 7 7 7 7 7 7 F F F 9 180-199 7 7 7 7 7 7 F F F 10 200-219 8 8 8 8 8 8 F F F 11 220-239 8 8 8 8 8 8 F F F 12 240-259 9 9 9 10 10 10 F F F 13 260-279 8 8 8 9 9 9 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 8 8 8 9 9 9 F F F 17 340-359 8 8 8 9 9 9 F F F 18 360-379 6 6 6 6 6 6 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 8 8 8 9 9 9 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 2 2 2 2 F F F 24 480-499 7 7 7 7 7 7 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 6 6 6 6 6 6 F F F 27 540-559 8 8 8 9 9 9 F F F 28 560-579 2 2 2 2 2 2 F F F 29 580-599 8 8 8 9 9 9 F F F Max gen: 9, Max flush: 10 Begin loop 5 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.469973506Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.515086594Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.515993115Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.51605126Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.516071698Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.516099166Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240} {"msg":"Crucible 28a2cee2-a386-4236-8db3-54b8c13412df has session id: 73f541b4-9827-4a24-a316-fd681d29af0a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.516613171Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Upstairs opts: Upstairs UUID: 28a2cee2-a386-4236-8db3-54b8c13412df, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.516669646Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Crucible stats registered with UUID: 28a2cee2-a386-4236-8db3-54b8c13412df","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.516699884Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:51.516726712Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.516941433Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"io task","client":"1","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.516990909Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"io task","client":"0","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517021106Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"io task","client":"2","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517123527Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"task":"crutest"} {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517156504Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240} {"msg":"28a2cee2-a386-4236-8db3-54b8c13412df active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.51719959Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517218858Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"client":"0","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517236497Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"client":"1","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517253626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"client":"2","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517275894Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"io task","client":"2","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517306431Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"io task","client":"0","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517326399Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"io task","client":"1","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"ds_connection connected from Ok(127.0.0.1:41116)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517513803Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"io task","client":"0","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"ds_connection connected from Ok(127.0.0.1:35924)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517558359Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"io task","client":"2","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"ds_connection connected from Ok(127.0.0.1:32804)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.517593766Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"io task","client":"1","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.518793881Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"client":"0","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.518842637Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"client":"2","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519242472Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"client":"1","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519435835Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519517828Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"[0]R flush_numbers[0..12]: [2, 9, 10, 9, 9, 9, 8, 8, 7, 7, 8, 8]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519747208Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"[0]R generation[0..12]: [2, 8, 9, 8, 8, 8, 8, 8, 7, 7, 8, 8]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519771206Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519791484Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"[1]R flush_numbers[0..12]: [2, 9, 10, 9, 9, 9, 8, 8, 7, 7, 8, 8]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519809973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"[1]R generation[0..12]: [2, 8, 9, 8, 8, 8, 8, 8, 7, 7, 8, 8]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519827981Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.51984728Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"[2]R flush_numbers[0..12]: [2, 9, 10, 9, 9, 9, 8, 8, 7, 7, 8, 8]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519865168Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"[2]R generation[0..12]: [2, 8, 9, 8, 8, 8, 8, 8, 7, 7, 8, 8]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519882966Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519902405Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519930272Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519948501Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Next flush: 11","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519964779Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519981098Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"":"downstairs","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.519998036Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.520013465Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"client":"0","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} Wait for a query_work_queue command to finish before sending IO {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.520045332Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"client":"1","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.52006465Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"client":"2","":"downstairs","session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.520083559Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.520100147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"28a2cee2-a386-4236-8db3-54b8c13412df is now active with session: 73f541b4-9827-4a24-a316-fd681d29af0a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.520118456Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.520133735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240,"session_id":"73f541b4-9827-4a24-a316-fd681d29af0a"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.520149603Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240} {"msg":"Activated sub_volume 28a2cee2-a386-4236-8db3-54b8c13412df","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:15:51.520163562Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 461 len 32768 data: 3 3 3 3 3 3 3 3 02/30 Read block 555 len 4096 03/30 Write block 201 len 16384 data: 3 2 2 2 04/30 Read block 522 len 40960 05/30 Flush 06/30 Write block 447 len 28672 data: 2 2 2 2 2 2 2 07/30 Write block 573 len 8192 data: 2 2 08/30 Flush 09/30 Write block 302 len 28672 data: 2 2 2 2 2 2 2 10/30 Write block 279 len 36864 data: 2 2 2 2 2 2 2 2 2 11/30 Read block 121 len 28672 12/30 Read block 383 len 24576 13/30 Read block 544 len 20480 14/30 Read block 26 len 16384 15/30 Write block 80 len 16384 data: 2 2 2 3 16/30 Write block 141 len 20480 data: 2 2 2 2 3 17/30 Read block 407 len 32768 18/30 Write block 495 len 12288 data: 3 3 3 19/30 Flush 20/30 Write block 58 len 16384 data: 3 3 3 2 21/30 Read block 538 len 4096 22/30 Write block 249 len 24576 data: 3 2 2 2 2 2 23/30 Read block 367 len 8192 24/30 Flush 25/30 Flush 26/30 Flush 27/30 Write block 231 len 24576 data: 2 2 2 2 2 2 28/30 Write block 569 len 24576 data: 2 3 3 2 3 3 29/30 Write block 568 len 8192 data: 2 3 30/30 Write block 23 len 28672 data: 2 2 2 2 2 2 3 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:22 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1008 Acked Write 7 Done Done Done false 1009 Acked Write 9 Done Done Done false 1010 Acked Read 7 Done Done Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Read 4 Done Sent Done false 1014 Acked Write 4 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Write 3 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Write 6 Done Sent Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 19 3 24 Done 20 3 19 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1007)) Some(JobId(1025)) Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:15:51.53808511Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1240} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 8 8 8 9 9 9 T F T <--- 2 040-059 10 9 10 14 10 14 F F F <--- 3 060-079 10 8 10 14 9 14 F F F <--- 4 080-099 10 8 10 13 9 13 F F F <--- 5 100-119 8 8 8 9 9 9 F F F 6 120-139 8 8 8 8 8 8 F F F 7 140-159 10 8 10 13 8 13 F F F <--- 8 160-179 7 7 7 7 7 7 F F F 9 180-199 7 7 7 7 7 7 F F F 10 200-219 10 10 10 11 11 11 F F F 11 220-239 8 8 8 8 8 8 T F T <--- 12 240-259 10 9 10 14 10 14 F F F <--- 13 260-279 10 8 10 13 9 13 F T F <--- 14 280-299 10 1 10 13 1 13 F T F <--- 15 300-319 10 1 10 13 1 13 F T F <--- 16 320-339 8 8 8 9 9 9 F F F 17 340-359 8 8 8 9 9 9 F F F 18 360-379 6 6 6 6 6 6 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 8 8 8 9 9 9 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 10 10 10 12 12 12 F F F 23 460-479 10 10 10 11 11 11 F F F 24 480-499 10 7 10 13 7 13 F F F <--- 25 500-519 7 7 7 7 7 7 F F F 26 520-539 6 6 6 6 6 6 F F F 27 540-559 8 8 8 9 9 9 F F F 28 560-579 10 10 10 12 12 12 T F T <--- 29 580-599 8 8 8 9 9 9 F F F Max gen: 10, Max flush: 14 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 11 11 11 15 15 15 F F F 2 040-059 10 10 10 14 14 14 F F F 3 060-079 10 10 10 14 14 14 F F F 4 080-099 10 10 10 13 13 13 F F F 5 100-119 8 8 8 9 9 9 F F F 6 120-139 8 8 8 8 8 8 F F F 7 140-159 10 10 10 13 13 13 F F F 8 160-179 7 7 7 7 7 7 F F F 9 180-199 7 7 7 7 7 7 F F F 10 200-219 10 10 10 11 11 11 F F F 11 220-239 11 11 11 15 15 15 F F F 12 240-259 10 10 10 14 14 14 F F F 13 260-279 10 10 10 13 13 13 F F F 14 280-299 10 10 10 13 13 13 F F F 15 300-319 10 10 10 13 13 13 F F F 16 320-339 8 8 8 9 9 9 F F F 17 340-359 8 8 8 9 9 9 F F F 18 360-379 6 6 6 6 6 6 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 8 8 8 9 9 9 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 10 10 10 12 12 12 F F F 23 460-479 10 10 10 11 11 11 F F F 24 480-499 10 10 10 13 13 13 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 6 6 6 6 6 6 F F F 27 540-559 8 8 8 9 9 9 F F F 28 560-579 11 11 11 15 15 15 F F F 29 580-599 8 8 8 9 9 9 F F F Max gen: 11, Max flush: 15 Begin loop 6 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:05.949787157Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:05.992868643Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:05.993311924Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.024150187Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.024827258Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.024879773Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.024899042Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.024926389Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248} {"msg":"Crucible c04a9697-fea4-4065-9410-2f8fdddab23b has session id: 1de77be9-a423-4585-a41f-f3c79d681ef9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.025507459Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Upstairs opts: Upstairs UUID: c04a9697-fea4-4065-9410-2f8fdddab23b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.025594521Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Crucible stats registered with UUID: c04a9697-fea4-4065-9410-2f8fdddab23b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.025630878Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:06.025650286Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.025829721Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.025878836Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.025909044Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.026085678Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"task":"crutest"} {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.026116436Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248} {"msg":"c04a9697-fea4-4065-9410-2f8fdddab23b active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.026145273Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.026161932Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"client":"0","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.02617964Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"client":"1","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.026196879Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"client":"2","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.026213517Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.026230806Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.026247964Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"ds_connection connected from Ok(127.0.0.1:58420)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.026331837Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"ds_connection connected from Ok(127.0.0.1:62337)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.026370383Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"ds_connection connected from Ok(127.0.0.1:51660)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.0264113Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.027583918Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"client":"1","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.027705487Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"client":"0","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.027841635Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"client":"2","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028170967Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028254849Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"[0]R flush_numbers[0..12]: [2, 15, 14, 14, 13, 9, 8, 13, 7, 7, 11, 15]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028369689Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"[0]R generation[0..12]: [2, 11, 10, 10, 10, 8, 8, 10, 7, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028393977Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028414585Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"[1]R flush_numbers[0..12]: [2, 15, 14, 14, 13, 9, 8, 13, 7, 7, 11, 15]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028433344Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"[1]R generation[0..12]: [2, 11, 10, 10, 10, 8, 8, 10, 7, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028452032Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.02847176Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"[2]R flush_numbers[0..12]: [2, 15, 14, 14, 13, 9, 8, 13, 7, 7, 11, 15]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028490409Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"[2]R generation[0..12]: [2, 11, 10, 10, 10, 8, 8, 10, 7, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028512887Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028532805Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028549404Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Generation requested: 12 >= found:12","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-12-24T00:16:06.028566482Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Next flush: 16","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.02858857Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028605139Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"":"downstairs","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028622397Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028637806Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"client":"0","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028654875Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"client":"1","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028671743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"client":"2","":"downstairs","session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028688981Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.02870519Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"c04a9697-fea4-4065-9410-2f8fdddab23b is now active with session: 1de77be9-a423-4585-a41f-f3c79d681ef9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028723389Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:16:06.028738837Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248,"session_id":"1de77be9-a423-4585-a41f-f3c79d681ef9"} {"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028754916Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248} {"msg":"Activated sub_volume c04a9697-fea4-4065-9410-2f8fdddab23b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:06.028768894Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 546 len 32768 data: 2 2 3 3 3 3 3 2 02/30 Read block 474 len 16384 03/30 Read block 239 len 16384 04/30 Read block 401 len 32768 05/30 Read block 281 len 4096 06/30 Read block 357 len 40960 07/30 Write block 170 len 28672 data: 2 2 2 3 3 3 4 08/30 Flush 09/30 Write block 121 len 36864 data: 2 2 2 2 2 2 2 2 2 10/30 Read block 124 len 12288 11/30 Write block 580 len 28672 data: 2 2 2 2 2 2 3 12/30 Read block 97 len 28672 13/30 Write block 103 len 36864 data: 3 3 3 3 3 3 2 2 2 14/30 Read block 448 len 20480 15/30 Write block 578 len 16384 data: 2 2 3 3 16/30 Read block 346 len 12288 17/30 Read block 343 len 16384 18/30 Flush 19/30 Write block 388 len 8192 data: 2 2 20/30 Read block 541 len 40960 21/30 Write block 470 len 28672 data: 2 2 2 2 2 2 2 22/30 Write block 256 len 8192 data: 2 3 23/30 Read block 118 len 24576 24/30 Read block 571 len 4096 25/30 Read block 417 len 16384 26/30 Read block 30 len 36864 27/30 Read block 263 len 36864 28/30 Write block 371 len 8192 data: 2 2 29/30 Write block 423 len 24576 data: 2 2 2 2 2 2 30/30 Write block 39 len 8192 data: 2 3 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 4 Sent Done Done false 1002 Acked Read 4 Sent Done Done false 1003 Acked Read 8 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Read 10 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Read 3 Sent Done Done false 1010 Acked Write 7 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Write 2 Sent Sent Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 3 2 34 Done 1 27 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:06.048417122Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1248} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 11 11 11 15 15 15 F T T <--- 2 040-059 10 10 10 14 14 14 F T T <--- 3 060-079 10 10 10 14 14 14 F F F 4 080-099 10 10 10 13 13 13 F F F 5 100-119 8 12 12 9 17 17 F F F <--- 6 120-139 8 12 12 8 17 17 F F F <--- 7 140-159 10 10 10 13 13 13 F F F 8 160-179 7 12 12 7 16 16 F F F <--- 9 180-199 7 7 7 7 7 7 F F F 10 200-219 10 10 10 11 11 11 F F F 11 220-239 11 11 11 15 15 15 F F F 12 240-259 10 10 10 14 14 14 F T T <--- 13 260-279 10 10 10 13 13 13 F F F 14 280-299 10 10 10 13 13 13 F F F 15 300-319 10 10 10 13 13 13 F F F 16 320-339 8 8 8 9 9 9 F F F 17 340-359 8 8 8 9 9 9 F F F 18 360-379 6 6 6 6 6 6 F T T <--- 19 380-399 2 2 2 2 2 2 F T T <--- 20 400-419 8 8 8 9 9 9 F F F 21 420-439 1 1 1 1 1 1 F T T <--- 22 440-459 10 10 10 12 12 12 F F F 23 460-479 10 10 10 11 11 11 F T T <--- 24 480-499 10 10 10 13 13 13 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 6 6 6 6 6 6 F F F 27 540-559 8 12 12 9 16 16 T F F <--- 28 560-579 11 12 12 15 17 17 F F F <--- 29 580-599 8 12 12 9 17 17 F F F <--- Max gen: 12, Max flush: 17 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 13 13 13 18 18 18 F F F 2 040-059 13 13 13 18 18 18 F F F 3 060-079 10 10 10 14 14 14 F F F 4 080-099 10 10 10 13 13 13 F F F 5 100-119 12 12 12 17 17 17 F F F 6 120-139 12 12 12 17 17 17 F F F 7 140-159 10 10 10 13 13 13 F F F 8 160-179 12 12 12 16 16 16 F F F 9 180-199 7 7 7 7 7 7 F F F 10 200-219 10 10 10 11 11 11 F F F 11 220-239 11 11 11 15 15 15 F F F 12 240-259 13 13 13 18 18 18 F F F 13 260-279 10 10 10 13 13 13 F F F 14 280-299 10 10 10 13 13 13 F F F 15 300-319 10 10 10 13 13 13 F F F 16 320-339 8 8 8 9 9 9 F F F 17 340-359 8 8 8 9 9 9 F F F 18 360-379 13 13 13 18 18 18 F F F 19 380-399 13 13 13 18 18 18 F F F 20 400-419 8 8 8 9 9 9 F F F 21 420-439 13 13 13 18 18 18 F F F 22 440-459 10 10 10 12 12 12 F F F 23 460-479 13 13 13 18 18 18 F F F 24 480-499 10 10 10 13 13 13 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 6 6 6 6 6 6 F F F 27 540-559 12 12 12 16 16 16 F F F 28 560-579 12 12 12 17 17 17 F F F 29 580-599 12 12 12 17 17 17 F F F Max gen: 13, Max flush: 18 Begin loop 7 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.48769455Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.532712597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.533504368Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.533551204Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.533567973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.533589981Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256} {"msg":"Crucible 2bfe9636-ffe2-4460-a56d-765a10ed38b3 has session id: 673ec9c6-e06c-4738-b01d-a44479ea3500","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.534190938Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Upstairs opts: Upstairs UUID: 2bfe9636-ffe2-4460-a56d-765a10ed38b3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.534259583Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Crucible stats registered with UUID: 2bfe9636-ffe2-4460-a56d-765a10ed38b3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.534297599Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:10.534329766Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.534540288Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.534595883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.53463425Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.534838082Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"task":"crutest"} {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.534879349Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256} {"msg":"2bfe9636-ffe2-4460-a56d-765a10ed38b3 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.534925854Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.534952952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"client":"0","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.534983829Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"client":"1","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.535014197Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"client":"2","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.535046904Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.535081331Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.535115978Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"ds_connection connected from Ok(127.0.0.1:50529)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.535152365Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"ds_connection connected from Ok(127.0.0.1:57131)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.535187271Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"ds_connection connected from Ok(127.0.0.1:35376)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.535219869Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.536494968Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"client":"0","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.536671012Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"client":"1","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.53680667Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"client":"2","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537054139Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537241663Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"[0]R flush_numbers[0..12]: [2, 18, 18, 14, 13, 17, 17, 13, 16, 7, 11, 15]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537346873Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"[0]R generation[0..12]: [2, 13, 13, 10, 10, 12, 12, 10, 12, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537376031Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537397059Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"[1]R flush_numbers[0..12]: [2, 18, 18, 14, 13, 17, 17, 13, 16, 7, 11, 15]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537420857Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"[1]R generation[0..12]: [2, 13, 13, 10, 10, 12, 12, 10, 12, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537452154Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537488291Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"[2]R flush_numbers[0..12]: [2, 18, 18, 14, 13, 17, 17, 13, 16, 7, 11, 15]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537522068Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"[2]R generation[0..12]: [2, 13, 13, 10, 10, 12, 12, 10, 12, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537551956Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537578073Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Max found gen is 14","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.5376123Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537644217Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Next flush: 19","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537664226Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537681524Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"":"downstairs","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537709482Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537738469Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"client":"0","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537763297Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"client":"1","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537804503Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"client":"2","":"downstairs","session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537836041Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537860759Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"2bfe9636-ffe2-4460-a56d-765a10ed38b3 is now active with session: 673ec9c6-e06c-4738-b01d-a44479ea3500","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537879367Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":"673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537903515Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256,"session_id":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "673ec9c6-e06c-4738-b01d-a44479ea3500"} {"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.537934022Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256} {"msg":"Activated sub_volume 2bfe9636-ffe2-4460-a56d-765a10ed38b3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:10.53795673Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 157 len 24576 data: 2 2 2 2 3 3 02/30 Read block 438 len 28672 03/30 Read block 291 len 8192 04/30 Read block 313 len 12288 05/30 Read block 510 len 28672 06/30 Flush 07/30 Write block 445 len 24576 data: 2 2 3 3 3 3 08/30 Flush 09/30 Write block 536 len 12288 data: 3 2 2 10/30 Read block 108 len 4096 11/30 Write block 341 len 12288 data: 2 2 2 12/30 Read block 557 len 4096 13/30 Read block 458 len 24576 14/30 Read block 310 len 16384 15/30 Write block 117 len 8192 data: 2 2 16/30 Flush 17/30 Write block 246 len 40960 data: 3 3 3 4 3 3 3 3 3 2 18/30 Read block 406 len 28672 19/30 Write block 64 len 12288 data: 2 2 2 20/30 Read block 108 len 8192 21/30 Write block 121 len 4096 data: 3 22/30 Write block 537 len 4096 data: 3 23/30 Read block 494 len 8192 24/30 Write block 550 len 32768 data: 4 4 4 3 2 2 2 2 25/30 Read block 585 len 20480 26/30 Read block 508 len 4096 27/30 Write block 433 len 4096 data: 2 28/30 Read block 563 len 36864 29/30 Read block 123 len 4096 30/30 Read block 323 len 24576 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Read 7 Done Done Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Read 1 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Read 4 Done Sent Done false 1014 Acked Write 2 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Write 1 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 0 25 Done 30 5 30 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) None Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:10.556346538Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1256} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 13 13 13 18 18 18 F F F 2 040-059 13 13 13 18 18 18 F F F 3 060-079 10 10 10 14 14 14 T F T <--- 4 080-099 10 10 10 13 13 13 F F F 5 100-119 14 12 14 21 17 21 F F F <--- 6 120-139 12 12 12 17 17 17 T F T <--- 7 140-159 14 10 14 19 13 19 F T F <--- 8 160-179 14 12 14 19 16 19 F T F <--- 9 180-199 7 7 7 7 7 7 F F F 10 200-219 10 10 10 11 11 11 F F F 11 220-239 11 11 11 15 15 15 F F F 12 240-259 13 13 13 18 18 18 T F T <--- 13 260-279 10 10 10 13 13 13 F F F 14 280-299 10 10 10 13 13 13 F F F 15 300-319 10 10 10 13 13 13 F F F 16 320-339 8 8 8 9 9 9 F F F 17 340-359 14 8 14 21 9 21 F F F <--- 18 360-379 13 13 13 18 18 18 F F F 19 380-399 13 13 13 18 18 18 F F F 20 400-419 8 8 8 9 9 9 F F F 21 420-439 13 13 13 18 18 18 T F T <--- 22 440-459 14 10 14 20 12 20 F F F <--- 23 460-479 13 13 13 18 18 18 F F F 24 480-499 10 10 10 13 13 13 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 14 6 14 21 6 21 T F T <--- 27 540-559 12 12 12 16 16 16 T F T <--- 28 560-579 12 12 12 17 17 17 F F F 29 580-599 12 12 12 17 17 17 F F F Max gen: 14, Max flush: 21 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 13 13 13 18 18 18 F F F 2 040-059 13 13 13 18 18 18 F F F 3 060-079 15 15 15 22 22 22 F F F 4 080-099 10 10 10 13 13 13 F F F 5 100-119 14 14 14 21 21 21 F F F 6 120-139 15 15 15 22 22 22 F F F 7 140-159 14 14 14 19 19 19 F F F 8 160-179 14 14 14 19 19 19 F F F 9 180-199 7 7 7 7 7 7 F F F 10 200-219 10 10 10 11 11 11 F F F 11 220-239 11 11 11 15 15 15 F F F 12 240-259 15 15 15 22 22 22 F F F 13 260-279 10 10 10 13 13 13 F F F 14 280-299 10 10 10 13 13 13 F F F 15 300-319 10 10 10 13 13 13 F F F 16 320-339 8 8 8 9 9 9 F F F 17 340-359 14 14 14 21 21 21 F F F 18 360-379 13 13 13 18 18 18 F F F 19 380-399 13 13 13 18 18 18 F F F 20 400-419 8 8 8 9 9 9 F F F 21 420-439 15 15 15 22 22 22 F F F 22 440-459 14 14 14 20 20 20 F F F 23 460-479 13 13 13 18 18 18 F F F 24 480-499 10 10 10 13 13 13 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 15 15 15 22 22 22 F F F 27 540-559 15 15 15 22 22 22 F F F 28 560-579 12 12 12 17 17 17 F F F 29 580-599 12 12 12 17 17 17 F F F Max gen: 15, Max flush: 22 Begin loop 8 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:14.954257649Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:14.99911919Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:14.999847676Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:14.999895892Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:14.999912471Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:14.999934189Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264} {"msg":"Crucible 6deffdc9-623c-4f7e-910b-7cc08da5a87e has session id: 6a32d7ca-e7de-4e17-ba88-fe400278403c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.000448324Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Upstairs opts: Upstairs UUID: 6deffdc9-623c-4f7e-910b-7cc08da5a87e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.000513038Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Crucible stats registered with UUID: 6deffdc9-623c-4f7e-910b-7cc08da5a87e","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.000548275Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:15.000574593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.000781715Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.000823711Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.000843589Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.001048292Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"task":"crutest"} {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.00107259Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264} {"msg":"6deffdc9-623c-4f7e-910b-7cc08da5a87e active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.001145153Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.00117782Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"client":"0","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.001200898Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"client":"1","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.001219007Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"client":"2","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.001240705Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.001272682Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.001322318Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"ds_connection connected from Ok(127.0.0.1:43852)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.001355795Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"ds_connection connected from Ok(127.0.0.1:47643)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.001387812Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"ds_connection connected from Ok(127.0.0.1:48636)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.001417859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.002524443Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"client":"2","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.002630584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"client":"0","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.00313598Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003171827Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003201514Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"client":"1","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"[0]R flush_numbers[0..12]: [2, 18, 18, 22, 13, 21, 22, 19, 19, 7, 11, 15]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003690751Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"[0]R generation[0..12]: [2, 13, 13, 15, 10, 14, 15, 14, 14, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003719599Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003740247Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"[1]R flush_numbers[0..12]: [2, 18, 18, 22, 13, 21, 22, 19, 19, 7, 11, 15]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003759265Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"[1]R generation[0..12]: [2, 13, 13, 15, 10, 14, 15, 14, 14, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003777664Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003797812Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"[2]R flush_numbers[0..12]: [2, 18, 18, 22, 13, 21, 22, 19, 19, 7, 11, 15]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.00381677Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"[2]R generation[0..12]: [2, 13, 13, 15, 10, 14, 15, 14, 14, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003834979Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:16:15.003864216Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003901513Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.00393357Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Next flush: 23","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003962158Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.003981406Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"":"downstairs","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.004010743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.004032912Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"client":"0","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.00405346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"client":"1","":"downstairs","session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.004082557Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.004115034Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.004140632Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"6deffdc9-623c-4f7e-910b-7cc08da5a87e is now active with session: 6a32d7ca-e7de-4e17-ba88-fe400278403c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.00415959Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.004175049Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264,"session_id":"6a32d7ca-e7de-4e17-ba88-fe400278403c"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.004191458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264} {"msg":"Activated sub_volume 6deffdc9-623c-4f7e-910b-7cc08da5a87e","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:15.004205626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 324 len 4096 data: 2 03/30 Write block 268 len 24576 data: 2 3 3 3 4 4 04/30 Write block 340 len 4096 data: 2 05/30 Read block 417 len 16384 06/30 Write block 324 len 32768 data: 3 2 2 2 2 2 3 3 07/30 Write block 445 len 12288 data: 3 3 4 08/30 Read block 253 len 20480 09/30 Read block 63 len 12288 10/30 Write block 538 len 20480 data: 3 2 2 2 2 11/30 Read block 520 len 32768 12/30 Write block 578 len 4096 data: 3 13/30 Read block 407 len 4096 14/30 Read block 476 len 40960 15/30 Read block 225 len 32768 16/30 Read block 145 len 20480 17/30 Write block 50 len 4096 data: 2 18/30 Write block 484 len 4096 data: 3 19/30 Write block 64 len 4096 data: 3 20/30 Write block 163 len 8192 data: 3 3 21/30 Write block 198 len 16384 data: 4 4 3 4 22/30 Write block 94 len 40960 data: 2 2 2 2 3 3 4 4 3 4 23/30 Write block 5 len 4096 data: 2 24/30 Read block 121 len 40960 25/30 Flush 26/30 Write block 222 len 36864 data: 3 3 4 4 3 3 3 2 2 27/30 Read block 353 len 32768 28/30 Read block 386 len 36864 29/30 Write block 384 len 16384 data: 2 2 2 2 30/30 Read block 569 len 36864 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 1 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Write 8 Done Sent Done false 1006 Acked Write 3 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 3 Done Sent Done false 1009 Acked Write 5 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 5 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Write 4 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Write 4 Done Sent Done false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 26 1 27 Done 29 3 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1000)) Some(JobId(1024)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:15.022117785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1264} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 2 16 24 2 24 F F F <--- 1 020-039 13 13 13 18 18 18 F F F 2 040-059 16 13 16 24 18 24 F F F <--- 3 060-079 16 15 16 24 22 24 F F F <--- 4 080-099 16 10 16 24 13 24 F F F <--- 5 100-119 16 14 16 24 21 24 F F F <--- 6 120-139 15 15 15 22 22 22 F F F 7 140-159 14 14 14 19 19 19 F F F 8 160-179 16 14 16 24 19 24 F F F <--- 9 180-199 16 7 16 24 7 24 F F F <--- 10 200-219 16 10 16 24 11 24 F F F <--- 11 220-239 11 11 11 15 15 15 T F T <--- 12 240-259 15 15 15 22 22 22 F F F 13 260-279 16 10 16 24 13 24 F T F <--- 14 280-299 10 10 10 13 13 13 F F F 15 300-319 10 10 10 13 13 13 F F F 16 320-339 16 8 16 24 9 24 F T F <--- 17 340-359 16 14 16 24 21 24 F T F <--- 18 360-379 13 13 13 18 18 18 F F F 19 380-399 13 13 13 18 18 18 T F T <--- 20 400-419 8 8 8 9 9 9 F F F 21 420-439 15 15 15 22 22 22 F F F 22 440-459 16 14 16 24 20 24 F F F <--- 23 460-479 13 13 13 18 18 18 F F F 24 480-499 16 10 16 24 13 24 F F F <--- 25 500-519 7 7 7 7 7 7 F F F 26 520-539 16 15 16 24 22 24 F F F <--- 27 540-559 16 15 16 24 22 24 F F F <--- 28 560-579 16 12 16 24 17 24 F F F <--- 29 580-599 12 12 12 17 17 17 F F F Max gen: 16, Max flush: 24 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 24 24 24 F F F 1 020-039 13 13 13 18 18 18 F F F 2 040-059 16 16 16 24 24 24 F F F 3 060-079 16 16 16 24 24 24 F F F 4 080-099 16 16 16 24 24 24 F F F 5 100-119 16 16 16 24 24 24 F F F 6 120-139 15 15 15 22 22 22 F F F 7 140-159 14 14 14 19 19 19 F F F 8 160-179 16 16 16 24 24 24 F F F 9 180-199 16 16 16 24 24 24 F F F 10 200-219 16 16 16 24 24 24 F F F 11 220-239 17 17 17 25 25 25 F F F 12 240-259 15 15 15 22 22 22 F F F 13 260-279 16 16 16 24 24 24 F F F 14 280-299 10 10 10 13 13 13 F F F 15 300-319 10 10 10 13 13 13 F F F 16 320-339 16 16 16 24 24 24 F F F 17 340-359 16 16 16 24 24 24 F F F 18 360-379 13 13 13 18 18 18 F F F 19 380-399 17 17 17 25 25 25 F F F 20 400-419 8 8 8 9 9 9 F F F 21 420-439 15 15 15 22 22 22 F F F 22 440-459 16 16 16 24 24 24 F F F 23 460-479 13 13 13 18 18 18 F F F 24 480-499 16 16 16 24 24 24 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 16 16 16 24 24 24 F F F 27 540-559 16 16 16 24 24 24 F F F 28 560-579 16 16 16 24 24 24 F F F 29 580-599 12 12 12 17 17 17 F F F Max gen: 17, Max flush: 25 Begin loop 9 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.450902736Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.497508515Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.498269558Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.498321584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.498338762Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.49836066Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272} {"msg":"Crucible 0d86765e-24f6-47c1-9335-53a9703af045 has session id: a49e388c-d9ba-48fe-abed-e5ef8424d9aa","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.498831719Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Upstairs opts: Upstairs UUID: 0d86765e-24f6-47c1-9335-53a9703af045, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.498887834Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Crucible stats registered with UUID: 0d86765e-24f6-47c1-9335-53a9703af045","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.498918732Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:19.498945029Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.49916272Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499214076Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499246733Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.4993982Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"task":"crutest"} {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499434597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272} {"msg":"0d86765e-24f6-47c1-9335-53a9703af045 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499461764Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499482252Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"client":"0","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.49950774Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"client":"1","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499529978Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"client":"2","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499550646Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499572065Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499605532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:56779)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499683275Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:56504)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499919534Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:47534)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.499961381Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.500933326Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"client":"2","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.500966483Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"client":"0","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501221421Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"client":"1","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501559531Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501600968Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"[0]R flush_numbers[0..12]: [24, 18, 24, 24, 24, 24, 22, 19, 24, 24, 24, 25]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501725257Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"[0]R generation[0..12]: [16, 13, 16, 16, 16, 16, 15, 14, 16, 16, 16, 17]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501749655Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501773433Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"[1]R flush_numbers[0..12]: [24, 18, 24, 24, 24, 24, 22, 19, 24, 24, 24, 25]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501792791Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"[1]R generation[0..12]: [16, 13, 16, 16, 16, 16, 15, 14, 16, 16, 16, 17]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.50181128Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501830898Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"[2]R flush_numbers[0..12]: [24, 18, 24, 24, 24, 24, 22, 19, 24, 24, 24, 25]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501849256Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"[2]R generation[0..12]: [16, 13, 16, 16, 16, 16, 15, 14, 16, 16, 16, 17]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501867685Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501887233Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Max found gen is 18","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501903552Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Generation requested: 18 >= found:18","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.50192053Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Next flush: 26","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501936598Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501952577Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"":"downstairs","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-12-24T00:16:19.501969566Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.501991504Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"client":"0","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.502008902Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"client":"1","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.502025721Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"client":"2","":"downstairs","session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.502044789Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.502061408Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"0d86765e-24f6-47c1-9335-53a9703af045 is now active with session: a49e388c-d9ba-48fe-abed-e5ef8424d9aa","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.502079356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.502094615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272,"session_id":"a49e388c-d9ba-48fe-abed-e5ef8424d9aa"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.502110574Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272} {"msg":"Activated sub_volume 0d86765e-24f6-47c1-9335-53a9703af045","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:19.502124522Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 43 len 8192 data: 4 4 02/30 Read block 67 len 36864 03/30 Read block 92 len 4096 04/30 Read block 98 len 28672 05/30 Read block 243 len 28672 06/30 Read block 107 len 16384 07/30 Read block 132 len 40960 08/30 Read block 290 len 32768 09/30 Read block 9 len 12288 10/30 Flush 11/30 Flush 12/30 Write block 443 len 32768 data: 2 2 4 4 5 4 4 4 13/30 Write block 432 len 20480 data: 2 3 2 2 2 14/30 Read block 213 len 40960 15/30 Flush 16/30 Write block 19 len 40960 data: 3 3 3 2 3 3 3 3 3 3 17/30 Read block 465 len 36864 18/30 Read block 180 len 16384 19/30 Read block 534 len 24576 20/30 Flush 21/30 Write block 180 len 4096 data: 4 22/30 Read block 411 len 20480 23/30 Write block 76 len 8192 data: 3 3 24/30 Read block 143 len 24576 25/30 Write block 148 len 16384 data: 2 2 2 2 26/30 Read block 180 len 4096 27/30 Write block 450 len 40960 data: 5 3 3 3 2 2 2 2 2 2 28/30 Read block 224 len 8192 29/30 Write block 236 len 32768 data: 3 2 2 2 2 2 2 2 30/30 Read block 326 len 24576 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Sent Done false 1001 Acked Read 9 Done Sent Done false 1002 Acked Read 1 Done Sent Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Read 7 Done Sent Done false 1005 Acked Read 4 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Read 3 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Write 5 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Read 6 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Write 8 Done Sent Done false 1029 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:19.521897019Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1272} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 18 16 18 29 24 29 F F F <--- 1 020-039 18 13 18 29 18 29 F F F <--- 2 040-059 18 16 18 26 24 26 F F F <--- 3 060-079 16 16 16 24 24 24 T F T <--- 4 080-099 16 16 16 24 24 24 F F F 5 100-119 16 16 16 24 24 24 F F F 6 120-139 15 15 15 22 22 22 F F F 7 140-159 14 14 14 19 19 19 T F T <--- 8 160-179 16 16 16 24 24 24 F F F 9 180-199 16 16 16 24 24 24 T F T <--- 10 200-219 16 16 16 24 24 24 F F F 11 220-239 17 17 17 25 25 25 T F T <--- 12 240-259 15 15 15 22 22 22 T F T <--- 13 260-279 16 16 16 24 24 24 F F F 14 280-299 10 10 10 13 13 13 F F F 15 300-319 10 10 10 13 13 13 F F F 16 320-339 16 16 16 24 24 24 F F F 17 340-359 16 16 16 24 24 24 F F F 18 360-379 13 13 13 18 18 18 F F F 19 380-399 17 17 17 25 25 25 F F F 20 400-419 8 8 8 9 9 9 F F F 21 420-439 18 15 18 28 22 28 F F F <--- 22 440-459 18 16 18 28 24 28 T F T <--- 23 460-479 13 13 13 18 18 18 F F F 24 480-499 16 16 16 24 24 24 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 16 16 16 24 24 24 F F F 27 540-559 16 16 16 24 24 24 F F F 28 560-579 16 16 16 24 24 24 F F F 29 580-599 12 12 12 17 17 17 F F F Max gen: 18, Max flush: 29 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 18 18 18 29 29 29 F F F 1 020-039 18 18 18 29 29 29 F F F 2 040-059 18 18 18 26 26 26 F F F 3 060-079 19 19 19 30 30 30 F F F 4 080-099 16 16 16 24 24 24 F F F 5 100-119 16 16 16 24 24 24 F F F 6 120-139 15 15 15 22 22 22 F F F 7 140-159 19 19 19 30 30 30 F F F 8 160-179 16 16 16 24 24 24 F F F 9 180-199 19 19 19 30 30 30 F F F 10 200-219 16 16 16 24 24 24 F F F 11 220-239 19 19 19 30 30 30 F F F 12 240-259 19 19 19 30 30 30 F F F 13 260-279 16 16 16 24 24 24 F F F 14 280-299 10 10 10 13 13 13 F F F 15 300-319 10 10 10 13 13 13 F F F 16 320-339 16 16 16 24 24 24 F F F 17 340-359 16 16 16 24 24 24 F F F 18 360-379 13 13 13 18 18 18 F F F 19 380-399 17 17 17 25 25 25 F F F 20 400-419 8 8 8 9 9 9 F F F 21 420-439 18 18 18 28 28 28 F F F 22 440-459 19 19 19 30 30 30 F F F 23 460-479 13 13 13 18 18 18 F F F 24 480-499 16 16 16 24 24 24 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 16 16 16 24 24 24 F F F 27 540-559 16 16 16 24 24 24 F F F 28 560-579 16 16 16 24 24 24 F F F 29 580-599 12 12 12 17 17 17 F F F Max gen: 19, Max flush: 30 Begin loop 10 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.92235226Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:23.965620489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.966029054Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.996846078Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.997480763Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.997526219Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.997542477Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.997564295Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280} {"msg":"Crucible 9b518077-c579-4d8d-88c4-a101b850a94f has session id: 2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998170563Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Upstairs opts: Upstairs UUID: 9b518077-c579-4d8d-88c4-a101b850a94f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998221308Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Crucible stats registered with UUID: 9b518077-c579-4d8d-88c4-a101b850a94f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998249506Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:23.998270864Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998507823Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998563868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998605545Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998768791Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"task":"crutest"} {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998795918Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280} {"msg":"9b518077-c579-4d8d-88c4-a101b850a94f active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998857643Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998877771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"client":"0","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.99889537Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"client":"1","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998912288Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"client":"2","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998928937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998946195Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.998972343Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"ds_connection connected from Ok(127.0.0.1:58271)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.99911741Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"ds_connection connected from Ok(127.0.0.1:50073)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.999161846Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"ds_connection connected from Ok(127.0.0.1:36507)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:23.999205073Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.000350723Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"client":"1","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.000510029Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"client":"2","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.000544416Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"client":"0","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001047192Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001153293Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"[0]R flush_numbers[0..12]: [29, 29, 26, 30, 24, 24, 22, 30, 24, 30, 24, 30]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001201719Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"[0]R generation[0..12]: [18, 18, 18, 19, 16, 16, 15, 19, 16, 19, 16, 19]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001224337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001244985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"[1]R flush_numbers[0..12]: [29, 29, 26, 30, 24, 24, 22, 30, 24, 30, 24, 30]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001263893Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"[1]R generation[0..12]: [18, 18, 18, 19, 16, 16, 15, 19, 16, 19, 16, 19]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001289151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001323868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"[2]R flush_numbers[0..12]: [29, 29, 26, 30, 24, 24, 22, 30, 24, 30, 24, 30]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001358745Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"[2]R generation[0..12]: [18, 18, 18, 19, 16, 16, 15, 19, 16, 19, 16, 19]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001389462Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001429299Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Max found gen is 20","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001451827Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Generation requested: 20 >= found:20","v":0,"name":"crucible","level":30,"time"Wait for a query_work_queue command to finish before sending IO :"2025-12-24T00:16:24.001478665Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Next flush: 31","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001514651Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001541949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"":"downstairs","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001560087Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001585755Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"client":"0","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001616693Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"client":"1","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.00164139Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"client":"2","":"downstairs","session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001664168Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001692046Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"9b518077-c579-4d8d-88c4-a101b850a94f is now active with session: 2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001724593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001742642Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280,"session_id":"2ae22a3a-0a0c-4c05-9dbf-3a98065ce84a"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001767149Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 } {"msg":"Activated sub_volume 9b518077-c579-4d8d-88c4-a101b850a94f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:24.001793777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 80 len 16384 data: 3 3 3 4 02/30 Read block 534 len 12288 03/30 Flush 04/30 Flush 05/30 Write block 276 len 8192 data: 4 5 06/30 Read block 565 len 16384 07/30 Flush 08/30 Write block 468 len 36864 data: 4 3 3 3 3 3 3 3 3 09/30 Read block 144 len 32768 10/30 Read block 261 len 4096 11/30 Read block 437 len 12288 12/30 Read block 228 len 36864 13/30 Write block 78 len 28672 data: 3 2 4 4 4 5 3 14/30 Write block 588 len 12288 data: 3 3 2 15/30 Read block 75 len 32768 16/30 Read block 106 len 28672 17/30 Read block 234 len 16384 18/30 Read block 578 len 16384 19/30 Write block 153 len 36864 data: 2 2 2 2 3 3 3 3 4 20/30 Read block 1 len 28672 21/30 Write block 481 len 4096 data: 2 22/30 Write block 248 len 40960 data: 4 5 4 4 4 4 4 3 3 4 23/30 Write block 299 len 16384 data: 2 2 2 3 24/30 Read block 374 len 12288 25/30 Write block 454 len 4096 data: 3 26/30 Write block 397 len 16384 data: 3 2 2 2 27/30 Write block 462 len 40960 data: 4 4 4 4 4 4 5 4 4 4 28/30 Read block 192 len 4096 29/30 Read block 553 len 16384 30/30 Write block 108 len 28672 data: 4 3 3 3 2 2 2 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:23 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1007 Acked Write 9 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Write 3 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 23 1 1 25 Done 0 22 22 44 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1006)) Some(JobId(1006)) Some(JobId(1006)) Downstairs last five completed: 1006 1005 1004 1003 1002 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:24.020567351Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1280} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 18 18 18 29 29 29 F F F 1 020-039 18 18 18 29 29 29 F F F 2 040-059 18 18 18 26 26 26 F F F 3 060-079 19 19 19 30 30 30 F T T <--- 4 080-099 20 20 20 31 31 31 F T T <--- 5 100-119 16 16 16 24 24 24 F T T <--- 6 120-139 15 15 15 22 22 22 F F F 7 140-159 19 19 19 30 30 30 F T T <--- 8 160-179 16 16 16 24 24 24 F T T <--- 9 180-199 19 19 19 30 30 30 F F F 10 200-219 16 16 16 24 24 24 F F F 11 220-239 19 19 19 30 30 30 F F F 12 240-259 19 19 19 30 30 30 F T T <--- 13 260-279 20 20 20 33 33 33 F F F 14 280-299 10 10 10 13 13 13 F T T <--- 15 300-319 10 10 10 13 13 13 F T T <--- 16 320-339 16 16 16 24 24 24 F F F 17 340-359 16 16 16 24 24 24 F F F 18 360-379 13 13 13 18 18 18 F F F 19 380-399 17 17 17 25 25 25 F T T <--- 20 400-419 8 8 8 9 9 9 F T T <--- 21 420-439 18 18 18 28 28 28 F F F 22 440-459 19 19 19 30 30 30 F T T <--- 23 460-479 13 13 13 18 18 18 F T T <--- 24 480-499 16 16 16 24 24 24 F T T <--- 25 500-519 7 7 7 7 7 7 F F F 26 520-539 16 16 16 24 24 24 F F F 27 540-559 16 16 16 24 24 24 F F F 28 560-579 16 16 16 24 24 24 F F F 29 580-599 12 12 12 17 17 17 F T T <--- Max gen: 20, Max flush: 33 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 18 18 18 29 29 29 F F F 1 020-039 18 18 18 29 29 29 F F F 2 040-059 18 18 18 26 26 26 F F F 3 060-079 21 21 21 34 34 34 F F F 4 080-099 21 21 21 34 34 34 F F F 5 100-119 21 21 21 34 34 34 F F F 6 120-139 15 15 15 22 22 22 F F F 7 140-159 21 21 21 34 34 34 F F F 8 160-179 21 21 21 34 34 34 F F F 9 180-199 19 19 19 30 30 30 F F F 10 200-219 16 16 16 24 24 24 F F F 11 220-239 19 19 19 30 30 30 F F F 12 240-259 21 21 21 34 34 34 F F F 13 260-279 20 20 20 33 33 33 F F F 14 280-299 21 21 21 34 34 34 F F F 15 300-319 21 21 21 34 34 34 F F F 16 320-339 16 16 16 24 24 24 F F F 17 340-359 16 16 16 24 24 24 F F F 18 360-379 13 13 13 18 18 18 F F F 19 380-399 21 21 21 34 34 34 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 18 18 18 28 28 28 F F F 22 440-459 21 21 21 34 34 34 F F F 23 460-479 21 21 21 34 34 34 F F F 24 480-499 21 21 21 34 34 34 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 16 16 16 24 24 24 F F F 27 540-559 16 16 16 24 24 24 F F F 28 560-579 16 16 16 24 24 24 F F F 29 580-599 21 21 21 34 34 34 F F F Max gen: 21, Max flush: 34 Begin loop 11 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.467259471Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.511943128Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.512683203Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.512741328Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.512765916Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.512793514Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288} {"msg":"Crucible 54659395-743a-4225-9108-b511e8b18fbe has session id: 09cf96d7-e928-493f-956f-086c1fa3a6b5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.513251474Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Upstairs opts: Upstairs UUID: 54659395-743a-4225-9108-b511e8b18fbe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.513365604Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Crucible stats registered with UUID: 54659395-743a-4225-9108-b511e8b18fbe","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.51340263Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:28.513430238Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.513651209Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.513700854Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.513734871Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.513886998Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"task":"crutest"} {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.513917545Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288} {"msg":"54659395-743a-4225-9108-b511e8b18fbe active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.51397597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.513997158Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"client":"0","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.514015277Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.514036785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"client":"2","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.514054064Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.514071612Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.514128467Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:28.514251766Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:28.514299062Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37306)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.514336159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:28.514374886Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"ds_connection connected from Ok(127.0.0.1:40153)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.514403093Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.514431871Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.514458458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.515479969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"client":"2","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.515564532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"client":"0","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.516150011Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:28.516244053Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.515720785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"ds_connection connected from Ok(127.0.0.1:50658)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.515831035Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.517783125Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"[0]R flush_numbers[0..12]: [29, 29, 26, 34, 34, 34, 22, 34, 34, 30, 24, 30]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518393072Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"[0]R generation[0..12]: [18, 18, 18, 21, 21, 21, 15, 21, 21, 19, 16, 19]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518423339Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518448507Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"[1]R flush_numbers[0..12]: [29, 29, 26, 34, 34, 34, 22, 34, 34, 30, 24, 30]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518471865Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"[1]R generation[0..12]: [18, 18, 18, 21, 21, 21, 15, 21, 21, 19, 16, 19]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518494903Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518519671Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"[2]R flush_numbers[0..12]: [29, 29, 26, 34, 34, 34, 22, 34, 34, 30, 24, 30]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518542749Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"[2]R generation[0..12]: [18, 18, 18, 21, 21, 21, 15, 21, 21, 19, 16, 19]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518565697Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518590095Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518610483Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518631901Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Next flush: 35","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:16:38.518658339Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518695026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"":"downstairs","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518716444Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518735542Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"client":"0","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.51875714Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"client":"1","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518779219Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"client":"2","":"downstairs","session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518801026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518821255Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"54659395-743a-4225-9108-b511e8b18fbe is now active with session: 09cf96d7-e928-493f-956f-086c1fa3a6b5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518843473Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518862541Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288,"session_id":"09cf96d7-e928-493f-956f-086c1fa3a6b5"} {"msg":"The guest has finished waiting for activation with:22","v":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518886249Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288} {"msg":"Activated sub_volume 54659395-743a-4225-9108-b511e8b18fbe","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:38.518904008Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 123 len 28672 data: 3 3 3 3 3 3 3 02/30 Read block 85 len 12288 03/30 Write block 361 len 16384 data: 4 3 3 3 04/30 Flush 05/30 Write block 431 len 12288 data: 2 3 4 06/30 Read block 548 len 32768 07/30 Write block 364 len 16384 data: 4 3 3 3 08/30 Read block 378 len 20480 09/30 Read block 509 len 36864 10/30 Read block 165 len 12288 11/30 Read block 76 len 36864 12/30 Write block 18 len 40960 data: 3 4 4 4 3 4 4 4 4 4 13/30 Read block 27 len 36864 14/30 Read block 268 len 32768 15/30 Write block 62 len 4096 data: 2 16/30 Write block 332 len 16384 data: 3 2 3 4 17/30 Write block 5 len 8192 data: 3 2 18/30 Read block 24 len 28672 19/30 Write block 197 len 28672 data: 3 5 5 4 5 3 3 20/30 Read block 443 len 20480 21/30 Read block 95 len 4096 22/30 Read block 230 len 36864 23/30 Write block 387 len 8192 data: 3 3 24/30 Write block 486 len 12288 data: 3 3 3 25/30 Write block 292 len 4096 data: 2 26/30 Read block 436 len 20480 27/30 Write block 539 len 20480 data: 3 3 3 3 2 28/30 Write block 372 len 16384 data: 3 2 2 2 29/30 Write block 301 len 20480 data: 3 4 3 3 3 30/30 Read block 224 len 16384 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 3 Done Done Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Write 4 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 9 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Read 9 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Write 1 Done Sent Done false 1015 Acked Write 4 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Write 5 Done Sent Sent false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 2 27 Done 26 1 24 51 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:38.537618647Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1288} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 18 18 18 29 29 29 T F T <--- 1 020-039 18 18 18 29 29 29 T F T <--- 2 040-059 18 18 18 26 26 26 F F F 3 060-079 21 21 21 34 34 34 T F T <--- 4 080-099 21 21 21 34 34 34 F F F 5 100-119 21 21 21 34 34 34 F F F 6 120-139 22 22 22 35 35 35 F F F 7 140-159 21 21 21 34 34 34 F F F 8 160-179 21 21 21 34 34 34 F F F 9 180-199 19 19 19 30 30 30 T F T <--- 10 200-219 16 16 16 24 24 24 T F T <--- 11 220-239 19 19 19 30 30 30 F F F 12 240-259 21 21 21 34 34 34 F F F 13 260-279 20 20 20 33 33 33 F F F 14 280-299 21 21 21 34 34 34 T F T <--- 15 300-319 21 21 21 34 34 34 T F T <--- 16 320-339 16 16 16 24 24 24 T F T <--- 17 340-359 16 16 16 24 24 24 F F F 18 360-379 22 22 22 35 35 35 T F T <--- 19 380-399 21 21 21 34 34 34 T F T <--- 20 400-419 21 21 21 34 34 34 F F F 21 420-439 18 18 18 28 28 28 T T T 22 440-459 21 21 21 34 34 34 F F F 23 460-479 21 21 21 34 34 34 F F F 24 480-499 21 21 21 34 34 34 T F T <--- 25 500-519 7 7 7 7 7 7 F F F 26 520-539 16 16 16 24 24 24 T F T <--- 27 540-559 16 16 16 24 24 24 T F T <--- 28 560-579 16 16 16 24 24 24 F F F 29 580-599 21 21 21 34 34 34 F F F Max gen: 22, Max flush: 35 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 36 36 36 F F F 1 020-039 23 23 23 36 36 36 F F F 2 040-059 18 18 18 26 26 26 F F F 3 060-079 23 23 23 36 36 36 F F F 4 080-099 21 21 21 34 34 34 F F F 5 100-119 21 21 21 34 34 34 F F F 6 120-139 22 22 22 35 35 35 F F F 7 140-159 21 21 21 34 34 34 F F F 8 160-179 21 21 21 34 34 34 F F F 9 180-199 23 23 23 36 36 36 F F F 10 200-219 23 23 23 36 36 36 F F F 11 220-239 19 19 19 30 30 30 F F F 12 240-259 21 21 21 34 34 34 F F F 13 260-279 20 20 20 33 33 33 F F F 14 280-299 23 23 23 36 36 36 F F F 15 300-319 23 23 23 36 36 36 F F F 16 320-339 23 23 23 36 36 36 F F F 17 340-359 16 16 16 24 24 24 F F F 18 360-379 23 23 23 36 36 36 F F F 19 380-399 23 23 23 36 36 36 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 23 23 23 36 36 36 F F F 22 440-459 21 21 21 34 34 34 F F F 23 460-479 21 21 21 34 34 34 F F F 24 480-499 23 23 23 36 36 36 F F F 25 500-519 7 7 7 7 7 7 F F F 26 520-539 23 23 23 36 36 36 F F F 27 540-559 23 23 23 36 36 36 F F F 28 560-579 16 16 16 24 24 24 F F F 29 580-599 21 21 21 34 34 34 F F F Max gen: 23, Max flush: 36 Begin loop 12 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:42.957443248Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:43.001118532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.001536265Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.032442562Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.033104254Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.03314979Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.033169509Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.033191527Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296} {"msg":"Crucible 9649612a-2d70-4ef8-af07-f9f814c3d5ca has session id: 7c820ad9-4f25-47a7-96e7-0375c5ad7c3d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.033630029Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Upstairs opts: Upstairs UUID: 9649612a-2d70-4ef8-af07-f9f814c3d5ca, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.033688894Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Crucible stats registered with UUID: 9649612a-2d70-4ef8-af07-f9f814c3d5ca","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.033716061Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:43.0337336Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.033950381Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.033988497Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034008346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.03419175Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"task":"crutest"} {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034227607Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296} {"msg":"9649612a-2d70-4ef8-af07-f9f814c3d5ca active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034295491Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034317729Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"client":"0","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034335607Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"client":"1","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034356875Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"client":"2","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034373994Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034391542Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034454587Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"ds_connection connected from Ok(127.0.0.1:51598)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034500333Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"ds_connection connected from Ok(127.0.0.1:53358)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.03453678Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"ds_connection connected from Ok(127.0.0.1:46572)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.034584616Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.035827637Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"client":"2","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.035915769Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"client":"1","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.035992383Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"client":"0","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.03637402Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036470791Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"[0]R flush_numbers[0..12]: [36, 36, 26, 36, 34, 34, 35, 34, 34, 36, 36, 30]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036568323Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"[0]R generation[0..12]: [23, 23, 18, 23, 21, 21, 22, 21, 21, 23, 23, 19]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.03659327Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036613969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"[1]R flush_numbers[0..12]: [36, 36, 26, 36, 34, 34, 35, 34, 34, 36, 36, 30]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036636447Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"[1]R generation[0..12]: [23, 23, 18, 23, 21, 21, 22, 21, 21, 23, 23, 19]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036655995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036675783Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"[2]R flush_numbers[0..12]: [36, 36, 26, 36, 34, 34, 35, 34, 34, 36, 36, 30]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036694392Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"[2]R generation[0..12]: [23, 23, 18, 23, 21, 21, 22, 21, 21, 23, 23, 19]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.03671294Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036733998Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036750777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036773745Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Next flush: 37","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036790143Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036806432Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"":"downstairs","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.03682372Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036839159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"client":"0","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036856167Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"client":"1","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036877356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"client":"2","":"downstairs","session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036898894Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036917962Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"9649612a-2d70-4ef8-af07-f9f814c3d5ca is now active with session: 7c820ad9-4f25-47a7-96e7-0375c5ad7c3d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.03693951Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036955089Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296,"session_id":"7c820ad9-4f25-47a7-96e7-0375c5ad7c3d"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036971417Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296} {"msg":"Activated sub_volume 9649612a-2d70-4ef8-af07-f9f814c3d5ca","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:43.036985716Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 347 len 16384 data: 3 3 3 3 02/30 Read block 542 len 24576 03/30 Flush 04/30 Flush 05/30 Write block 217 len 24576 data: 3 3 3 3 3 4 06/30 Write block 562 len 4096 data: 2 07/30 Read block 412 len 8192 08/30 Read block 509 len 4096 09/30 Flush 10/30 Flush 11/30 Flush 12/30 Write block 491 len 4096 data: 3 13/30 Write block 509 len 36864 data: 3 2 2 2 2 2 2 2 2 14/30 Read block 263 len 24576 15/30 Read block 411 len 32768 16/30 Write block 296 len 36864 data: 2 2 2 3 3 4 5 4 4 17/30 Write block 83 len 20480 data: 6 4 3 3 3 18/30 Read block 517 len 12288 19/30 Read block 382 len 8192 20/30 Read block 523 len 4096 21/30 Write block 127 len 36864 data: 4 4 4 2 2 2 3 3 4 22/30 Write block 358 len 12288 data: 4 4 4 23/30 Write block 23 len 20480 data: 5 5 5 5 5 24/30 Write block 586 len 24576 data: 4 4 4 4 3 2 25/30 Flush 26/30 Flush 27/30 Write block 587 len 4096 data: 5 28/30 Read block 445 len 20480 29/30 Read block 469 len 12288 30/30 Read block 83 len 28672 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:19 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1011 Acked Write 1 Done Done Done false 1012 Acked Write 9 Done Done Done false 1013 Acked Read 6 Done Done Done false 1014 Acked Read 8 Done Done Done false 1015 Acked Write 9 Done Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Write 6 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Read 7 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 14 0 0 14 Done 5 19 19 43 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: 1010 1009 1008 1007 1006 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:43.05450029Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1296} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 36 36 36 F F F 1 020-039 23 24 24 36 42 42 F F F <--- 2 040-059 18 18 18 26 26 26 F F F 3 060-079 23 23 23 36 36 36 F F F 4 080-099 21 24 24 34 42 42 F F F <--- 5 100-119 21 21 21 34 34 34 F F F 6 120-139 22 24 24 35 42 42 F F F <--- 7 140-159 21 21 21 34 34 34 F F F 8 160-179 21 21 21 34 34 34 F F F 9 180-199 23 23 23 36 36 36 F F F 10 200-219 24 24 24 39 39 39 F F F 11 220-239 24 24 24 39 39 39 F F F 12 240-259 21 21 21 34 34 34 F F F 13 260-279 20 20 20 33 33 33 F F F 14 280-299 23 24 24 36 42 42 T F F <--- 15 300-319 23 24 24 36 42 42 T F F <--- 16 320-339 23 23 23 36 36 36 F F F 17 340-359 24 24 24 37 42 42 F F F <--- 18 360-379 23 24 24 36 42 42 F F F <--- 19 380-399 23 23 23 36 36 36 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 23 23 23 36 36 36 F F F 22 440-459 21 21 21 34 34 34 F F F 23 460-479 21 21 21 34 34 34 F F F 24 480-499 23 24 24 36 42 42 T F F <--- 25 500-519 7 24 24 7 42 42 T F F <--- 26 520-539 23 23 23 36 36 36 F F F 27 540-559 23 23 23 36 36 36 F F F 28 560-579 24 24 24 39 39 39 F F F 29 580-599 21 24 24 34 42 42 F T T <--- Max gen: 24, Max flush: 42 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 36 36 36 F F F 1 020-039 24 24 24 42 42 42 F F F 2 040-059 18 18 18 26 26 26 F F F 3 060-079 23 23 23 36 36 36 F F F 4 080-099 24 24 24 42 42 42 F F F 5 100-119 21 21 21 34 34 34 F F F 6 120-139 24 24 24 42 42 42 F F F 7 140-159 21 21 21 34 34 34 F F F 8 160-179 21 21 21 34 34 34 F F F 9 180-199 23 23 23 36 36 36 F F F 10 200-219 24 24 24 39 39 39 F F F 11 220-239 24 24 24 39 39 39 F F F 12 240-259 21 21 21 34 34 34 F F F 13 260-279 20 20 20 33 33 33 F F F 14 280-299 24 24 24 42 42 42 F F F 15 300-319 24 24 24 42 42 42 F F F 16 320-339 23 23 23 36 36 36 F F F 17 340-359 24 24 24 42 42 42 F F F 18 360-379 24 24 24 42 42 42 F F F 19 380-399 23 23 23 36 36 36 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 23 23 23 36 36 36 F F F 22 440-459 21 21 21 34 34 34 F F F 23 460-479 21 21 21 34 34 34 F F F 24 480-499 24 24 24 42 42 42 F F F 25 500-519 24 24 24 42 42 42 F F F 26 520-539 23 23 23 36 36 36 F F F 27 540-559 23 23 23 36 36 36 F F F 28 560-579 24 24 24 39 39 39 F F F 29 580-599 25 25 25 43 43 43 F F F Max gen: 25, Max flush: 43 Begin loop 13 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.477382304Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:47.521594752Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.522009906Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.55317361Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.553840842Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.553886968Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.553902836Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.553925124Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304} {"msg":"Crucible 3872782e-794f-4638-8bae-56ef674749f2 has session id: b5f449f7-9aa0-4d7a-b95d-337f98b626df","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.554601215Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Upstairs opts: Upstairs UUID: 3872782e-794f-4638-8bae-56ef674749f2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.55467034Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Crucible stats registered with UUID: 3872782e-794f-4638-8bae-56ef674749f2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.554699977Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:47.554720865Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.554902369Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.554943796Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.554965114Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555135719Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"task":"crutest"} {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555186684Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304} {"msg":"3872782e-794f-4638-8bae-56ef674749f2 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555227911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555246009Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"client":"0","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555264258Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"client":"1","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555281416Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"client":"2","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555298175Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555320793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.55534865Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"ds_connection connected from Ok(127.0.0.1:36197)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555604848Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"ds_connection connected from Ok(127.0.0.1:41571)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555650334Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"ds_connection connected from Ok(127.0.0.1:44277)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.555685771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.556689573Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"client":"1","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.556724861Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"client":"2","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.556796884Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"client":"0","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557292661Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557400722Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"[0]R flush_numbers[0..12]: [36, 42, 26, 36, 42, 34, 42, 34, 34, 36, 39, 39]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557437538Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"[0]R generation[0..12]: [23, 24, 18, 23, 24, 21, 24, 21, 21, 23, 24, 24]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557457717Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557477795Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"[1]R flush_numbers[0..12]: [36, 42, 26, 36, 42, 34, 42, 34, 34, 36, 39, 39]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557496533Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"[1]R generation[0..12]: [23, 24, 18, 23, 24, 21, 24, 21, 21, 23, 24, 24]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557514992Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.55753473Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"[2]R flush_numbers[0..12]: [36, 42, 26, 36, 42, 34, 42, 34, 34, 36, 39, 39]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557553348Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"[2]R generation[0..12]: [23, 24, 18, 23, 24, 21, 24, 21, 21, 23, 24, 24]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557571657Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557591095Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557607713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557624842Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Next flush: 44","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.55764445Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} Wait for a query_work_queue command to finish before sending IO {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557677098Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"":"downstairs","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557694916Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557710475Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"client":"0","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557727533Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"client":"1","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557744672Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"client":"2","":"downstairs","session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.55776283Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557779458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"3872782e-794f-4638-8bae-56ef674749f2 is now active with session: b5f449f7-9aa0-4d7a-b95d-337f98b626df","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557797937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557814435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304,"session_id":"b5f449f7-9aa0-4d7a-b95d-337f98b626df"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557830694Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304} {"msg":"Activated sub_volume 3872782e-794f-4638-8bae-56ef674749f2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:47.557844793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 530 len 20480 data: 2 2 2 2 3 02/30 Write block 421 len 32768 data: 2 2 3 3 3 3 3 3 03/30 Write block 75 len 12288 data: 4 4 4 04/30 Write block 181 len 36864 data: 4 4 3 2 3 3 3 3 3 05/30 Read block 230 len 4096 06/30 Read block 151 len 32768 07/30 Read block 278 len 32768 08/30 Flush 09/30 Write block 186 len 16384 data: 4 4 4 4 10/30 Read block 549 len 28672 11/30 Read block 428 len 36864 12/30 Read block 18 len 8192 13/30 Write block 341 len 20480 data: 3 3 3 2 2 14/30 Write block 247 len 20480 data: 4 5 6 5 5 15/30 Flush 16/30 Write block 446 len 8192 data: 5 6 17/30 Read block 354 len 40960 18/30 Read block 506 len 28672 19/30 Read block 6 len 24576 20/30 Write block 92 len 24576 data: 2 2 3 3 3 3 21/30 Read block 229 len 28672 22/30 Read block 86 len 20480 23/30 Read block 482 len 24576 24/30 Write block 198 len 8192 data: 6 6 25/30 Read block 310 len 16384 26/30 Write block 515 len 12288 data: 3 3 3 27/30 Read block 553 len 40960 28/30 Write block 503 len 28672 data: 2 2 3 3 2 2 4 29/30 Read block 80 len 12288 30/30 Read block 63 len 40960 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Sent Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Write 9 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 10 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Read 5 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Read 10 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 0 31 Done 0 29 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:47.577142521Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1304} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 36 36 36 F F F 1 020-039 24 24 24 42 42 42 F F F 2 040-059 18 18 18 26 26 26 F F F 3 060-079 23 26 26 36 44 44 F F F <--- 4 080-099 24 24 24 42 42 42 F T T <--- 5 100-119 21 21 21 34 34 34 F F F 6 120-139 24 24 24 42 42 42 F F F 7 140-159 21 21 21 34 34 34 F F F 8 160-179 21 21 21 34 34 34 F F F 9 180-199 23 26 26 36 45 45 F T T <--- 10 200-219 24 24 24 39 39 39 F F F 11 220-239 24 24 24 39 39 39 F F F 12 240-259 21 26 26 34 45 45 F F F <--- 13 260-279 20 20 20 33 33 33 F F F 14 280-299 24 24 24 42 42 42 F F F 15 300-319 24 24 24 42 42 42 F F F 16 320-339 23 23 23 36 36 36 F F F 17 340-359 24 26 26 42 45 45 F F F <--- 18 360-379 24 24 24 42 42 42 F F F 19 380-399 23 23 23 36 36 36 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 23 26 26 36 44 44 F F F <--- 22 440-459 21 21 21 34 34 34 F T T <--- 23 460-479 21 21 21 34 34 34 F F F 24 480-499 24 24 24 42 42 42 F F F 25 500-519 24 24 24 42 42 42 F T T <--- 26 520-539 23 26 26 36 44 44 F F F <--- 27 540-559 23 23 23 36 36 36 F F F 28 560-579 24 24 24 39 39 39 F F F 29 580-599 25 25 25 43 43 43 F F F Max gen: 26, Max flush: 45 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 13 Verifying data now No change after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 36 36 36 F F F 1 020-039 24 24 24 42 42 42 F F F 2 040-059 18 18 18 26 26 26 F F F 3 060-079 26 26 26 44 44 44 F F F 4 080-099 27 27 27 46 46 46 F F F 5 100-119 21 21 21 34 34 34 F F F 6 120-139 24 24 24 42 42 42 F F F 7 140-159 21 21 21 34 34 34 F F F 8 160-179 21 21 21 34 34 34 F F F 9 180-199 27 27 27 46 46 46 F F F 10 200-219 24 24 24 39 39 39 F F F 11 220-239 24 24 24 39 39 39 F F F 12 240-259 26 26 26 45 45 45 F F F 13 260-279 20 20 20 33 33 33 F F F 14 280-299 24 24 24 42 42 42 F F F 15 300-319 24 24 24 42 42 42 F F F 16 320-339 23 23 23 36 36 36 F F F 17 340-359 26 26 26 45 45 45 F F F 18 360-379 24 24 24 42 42 42 F F F 19 380-399 23 23 23 36 36 36 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 26 26 26 44 44 44 F F F 22 440-459 27 27 27 46 46 46 F F F 23 460-479 21 21 21 34 34 34 F F F 24 480-499 24 24 24 42 42 42 F F F 25 500-519 27 27 27 46 46 46 F F F 26 520-539 26 26 26 44 44 44 F F F 27 540-559 23 23 23 36 36 36 F F F 28 560-579 24 24 24 39 39 39 F F F 29 580-599 25 25 25 43 43 43 F F F Max gen: 27, Max flush: 46 Begin loop 14 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:51.996085799Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.041459785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.04232225Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.042369936Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.042386714Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.042412092Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312} {"msg":"Crucible 97fbc0b1-aa70-478c-88a2-184809f172fc has session id: 2c6e042e-113b-40fc-aabc-ff9bb2ac1496","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.042911008Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Upstairs opts: Upstairs UUID: 97fbc0b1-aa70-478c-88a2-184809f172fc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.042966204Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Crucible stats registered with UUID: 97fbc0b1-aa70-478c-88a2-184809f172fc","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.042996371Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:52.043023149Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043250309Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043292985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043313433Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043538664Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"task":"crutest"} {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043574441Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312} {"msg":"97fbc0b1-aa70-478c-88a2-184809f172fc active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043644224Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043665953Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"client":"0","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043683631Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"client":"1","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.04370065Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"client":"2","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043717418Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043738936Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043760105Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"ds_connection connected from Ok(127.0.0.1:35139)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043907152Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"ds_connection connected from Ok(127.0.0.1:47669)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043949198Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"ds_connection connected from Ok(127.0.0.1:41480)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.043983855Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.045220107Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"client":"2","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.045271243Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"client":"0","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.045793607Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.045926206Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.046733555Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"client":"1","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"[0]R flush_numbers[0..12]: [36, 42, 26, 44, 46, 34, 42, 34, 34, 46, 39, 39]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047241591Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"[0]R generation[0..12]: [23, 24, 18, 26, 27, 21, 24, 21, 21, 27, 24, 24]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047267589Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047288557Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"[1]R flush_numbers[0..12]: [36, 42, 26, 44, 46, 34, 42, 34, 34, 46, 39, 39]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047307245Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"[1]R generation[0..12]: [23, 24, 18, 26, 27, 21, 24, 21, 21, 27, 24, 24]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047325904Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047349892Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"[2]R flush_numbers[0..12]: [36, 42, 26, 44, 46, 34, 42, 34, 34, 46, 39, 39]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.04736871Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"[2]R generation[0..12]: [23, 24, 18, 26, 27, 21, 24, 21, 21, 27, 24, 24]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047387348Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047407707Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Max found gen is 28","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047424455Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Generation requested: 28 >= found:28","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047441554Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Next flush: 47","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047457852Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047474071Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"":"downstairs","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047491279Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047506688Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"client":"0","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Wait for a query_work_queue command to finish before sending IO Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047530006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"client":"1","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047546735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"client":"2","":"downstairs","session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047564213Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047580632Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"97fbc0b1-aa70-478c-88a2-184809f172fc is now active with session: 2c6e042e-113b-40fc-aabc-ff9bb2ac1496","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.04759853Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047615809Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312,"session_id":"2c6e042e-113b-40fc-aabc-ff9bb2ac1496"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047632147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312} {"msg":"Activated sub_volume 97fbc0b1-aa70-478c-88a2-184809f172fc","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:52.047646076Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 39 len 16384 data: 3 4 3 4 02/30 Write block 325 len 20480 data: 3 3 3 3 3 03/30 Read block 485 len 8192 04/30 Flush 05/30 Write block 10 len 4096 data: 2 06/30 Write block 288 len 32768 data: 2 2 2 2 3 2 2 2 07/30 Read block 146 len 16384 08/30 Write block 355 len 12288 data: 5 5 5 09/30 Read block 567 len 32768 10/30 Read block 344 len 16384 11/30 Write block 384 len 8192 data: 3 3 12/30 Read block 166 len 36864 13/30 Read block 53 len 16384 14/30 Write block 97 len 16384 data: 4 4 4 5 15/30 Read block 172 len 4096 16/30 Flush 17/30 Write block 522 len 8192 data: 2 2 18/30 Read block 407 len 20480 19/30 Flush 20/30 Write block 290 len 32768 data: 3 3 4 3 3 3 3 3 21/30 Write block 495 len 32768 data: 4 4 4 3 3 3 3 2 22/30 Write block 242 len 12288 data: 3 3 2 23/30 Write block 444 len 32768 data: 3 5 6 7 5 5 6 4 24/30 Read block 63 len 4096 25/30 Read block 349 len 36864 26/30 Read block 211 len 28672 27/30 Write block 76 len 20480 data: 5 5 4 3 5 28/30 Write block 108 len 24576 data: 5 4 4 4 3 3 29/30 Write block 455 len 20480 data: 3 3 3 3 3 30/30 Read block 237 len 24576 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 5 Done Sent Done false 1002 Acked Read 2 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 1 Done Sent Done false 1005 Acked Write 8 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Write 3 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 1 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Write 5 Done Sent Done false 1029 Acked Read 6 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 0 30 Done 29 1 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) None Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:52.065434626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1312} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 23 28 48 36 48 F F F <--- 1 020-039 28 24 28 47 42 47 F T F <--- 2 040-059 28 18 28 47 26 47 F T F <--- 3 060-079 26 26 26 44 44 44 T F T <--- 4 080-099 28 27 28 48 46 48 T F T <--- 5 100-119 28 21 28 48 34 48 T F T <--- 6 120-139 24 24 24 42 42 42 F F F 7 140-159 21 21 21 34 34 34 F F F 8 160-179 21 21 21 34 34 34 F F F 9 180-199 27 27 27 46 46 46 F F F 10 200-219 24 24 24 39 39 39 F F F 11 220-239 24 24 24 39 39 39 F F F 12 240-259 26 26 26 45 45 45 T F T <--- 13 260-279 20 20 20 33 33 33 F F F 14 280-299 28 24 28 48 42 48 T F T <--- 15 300-319 24 24 24 42 42 42 F F F 16 320-339 28 23 28 47 36 47 F F F <--- 17 340-359 28 26 28 48 45 48 F F F <--- 18 360-379 24 24 24 42 42 42 F F F 19 380-399 28 23 28 48 36 48 F F F <--- 20 400-419 21 21 21 34 34 34 F F F 21 420-439 26 26 26 44 44 44 F F F 22 440-459 27 27 27 46 46 46 T F T <--- 23 460-479 21 21 21 34 34 34 F F F 24 480-499 24 24 24 42 42 42 T F T <--- 25 500-519 27 27 27 46 46 46 T F T <--- 26 520-539 28 26 28 49 44 49 F F F <--- 27 540-559 23 23 23 36 36 36 F F F 28 560-579 24 24 24 39 39 39 F F F 29 580-599 25 25 25 43 43 43 F F F Max gen: 28, Max flush: 49 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 48 48 48 F F F 1 020-039 28 28 28 47 47 47 F F F 2 040-059 28 28 28 47 47 47 F F F 3 060-079 29 29 29 50 50 50 F F F 4 080-099 29 29 29 50 50 50 F F F 5 100-119 29 29 29 50 50 50 F F F 6 120-139 24 24 24 42 42 42 F F F 7 140-159 21 21 21 34 34 34 F F F 8 160-179 21 21 21 34 34 34 F F F 9 180-199 27 27 27 46 46 46 F F F 10 200-219 24 24 24 39 39 39 F F F 11 220-239 24 24 24 39 39 39 F F F 12 240-259 29 29 29 50 50 50 F F F 13 260-279 20 20 20 33 33 33 F F F 14 280-299 29 29 29 50 50 50 F F F 15 300-319 24 24 24 42 42 42 F F F 16 320-339 28 28 28 47 47 47 F F F 17 340-359 28 28 28 48 48 48 F F F 18 360-379 24 24 24 42 42 42 F F F 19 380-399 28 28 28 48 48 48 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 26 26 26 44 44 44 F F F 22 440-459 29 29 29 50 50 50 F F F 23 460-479 21 21 21 34 34 34 F F F 24 480-499 29 29 29 50 50 50 F F F 25 500-519 29 29 29 50 50 50 F F F 26 520-539 28 28 28 49 49 49 F F F 27 540-559 23 23 23 36 36 36 F F F 28 560-579 24 24 24 39 39 39 F F F 29 580-599 25 25 25 43 43 43 F F F Max gen: 29, Max flush: 50 Begin loop 15 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.481864794Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:56.525696144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.526095869Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.55752879Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.558379726Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.558430001Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.55844923Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.558476397Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320} {"msg":"Crucible 896efdb8-112e-4b05-ba49-f563eb19e4d3 has session id: 55dbfa0f-ee7b-448d-bb67-01917e199a6d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.558972944Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Upstairs opts: Upstairs UUID: 896efdb8-112e-4b05-ba49-f563eb19e4d3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559045128Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Crucible stats registered with UUID: 896efdb8-112e-4b05-ba49-f563eb19e4d3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559072095Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:56.559092453Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559239041Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559292046Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559316684Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559393427Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"task":"crutest"} {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559413066Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320} {"msg":"896efdb8-112e-4b05-ba49-f563eb19e4d3 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559470571Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559505128Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"client":"0","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559537725Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"client":"1","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559572232Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"client":"2","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559606629Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559643876Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559676923Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:59249)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.55970754Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:61354)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.559740207Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:59455)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.55981721Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561038574Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"client":"2","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561089699Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"client":"1","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561184721Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"client":"0","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.56166338Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561761211Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"[0]R flush_numbers[0..12]: [48, 47, 47, 50, 50, 50, 42, 34, 34, 46, 39, 39]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561829515Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"[0]R generation[0..12]: [28, 28, 28, 29, 29, 29, 24, 21, 21, 27, 24, 24]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561853713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561873831Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"[1]R flush_numbers[0..12]: [48, 47, 47, 50, 50, 50, 42, 34, 34, 46, 39, 39]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.56189256Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"[1]R generation[0..12]: [28, 28, 28, 29, 29, 29, 24, 21, 21, 27, 24, 24]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561911228Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561930866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"[2]R flush_numbers[0..12]: [48, 47, 47, 50, 50, 50, 42, 34, 34, 46, 39, 39]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561949555Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"[2]R generation[0..12]: [28, 28, 28, 29, 29, 29, 24, 21, 21, 27, 24, 24]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561968223Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.561987741Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.56200421Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562027888Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Next flush: 51","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562058645Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562075604Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"":"downstairs","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562099262Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.56211453Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"client":"0","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562131449Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"client":"1","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562148217Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"client":"2","":"downstairs","session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562165336Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562181734Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"896efdb8-112e-4b05-ba49-f563eb19e4d3 is now active with session: 55dbfa0f-ee7b-448d-bb67-01917e199a6d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562199713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562215452Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320,"session_id":"55dbfa0f-ee7b-448d-bb67-01917e199a6d"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.56223164Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320} {"msg":"Activated sub_volume 896efdb8-112e-4b05-ba49-f563eb19e4d3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:16:56.562245819Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 426 len 36864 data: 4 4 4 2 2 3 4 5 3 02/30 Read block 477 len 20480 03/30 Write block 499 len 12288 data: 4 4 4 04/30 Write block 525 len 4096 data: 2 05/30 Read block 304 len 32768 06/30 Flush 07/30 Write block 470 len 4096 data: 5 08/30 Write block 75 len 40960 data: 5 6 6 5 4 6 5 5 7 5 09/30 Flush 10/30 Write block 480 len 16384 data: 2 3 3 3 11/30 Flush 12/30 Write block 151 len 12288 data: 3 2 3 13/30 Flush 14/30 Write block 267 len 24576 data: 2 3 4 4 4 5 15/30 Write block 220 len 20480 data: 4 4 5 4 5 16/30 Write block 484 len 36864 data: 4 3 4 4 4 3 3 4 2 17/30 Read block 541 len 12288 18/30 Read block 335 len 8192 19/30 Read block 508 len 4096 20/30 Write block 255 len 12288 data: 4 4 5 21/30 Write block 530 len 36864 data: 3 3 3 3 4 3 4 4 4 22/30 Write block 182 len 24576 data: 5 4 3 4 5 5 23/30 Read block 120 len 28672 24/30 Read block 120 len 4096 25/30 Write block 63 len 24576 data: 2 4 3 3 2 3 26/30 Read block 19 len 40960 27/30 Read block 98 len 4096 28/30 Write block 113 len 40960 data: 4 3 2 2 3 3 2 2 4 3 29/30 Write block 471 len 20480 data: 5 4 4 4 4 30/30 Read block 79 len 24576 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:24 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1006 Acked Write 1 Done Done Done false 1007 Acked Write 10 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 6 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Read 2 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Write 3 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 10 Sent Done Done false 1028 Acked Write 5 Sent Done Done false 1029 Acked Read 6 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 23 1 0 24 Done 1 23 24 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1005)) Some(JobId(1012)) Some(JobId(1012)) Downstairs last five completed: 1005 1004 1003 1002 1001 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:16:56.581035911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1320} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 48 48 48 F F F 1 020-039 28 28 28 47 47 47 F F F 2 040-059 28 28 28 47 47 47 F F F 3 060-079 29 30 30 50 52 52 F T T <--- 4 080-099 29 30 30 50 52 52 F F F <--- 5 100-119 29 29 29 50 50 50 F T T <--- 6 120-139 24 24 24 42 42 42 F T T <--- 7 140-159 21 30 30 34 54 54 F F F <--- 8 160-179 21 21 21 34 34 34 F F F 9 180-199 27 27 27 46 46 46 F T T <--- 10 200-219 24 24 24 39 39 39 F F F 11 220-239 24 24 24 39 39 39 F T T <--- 12 240-259 29 29 29 50 50 50 F T T <--- 13 260-279 20 20 20 33 33 33 F T T <--- 14 280-299 29 29 29 50 50 50 F F F 15 300-319 24 24 24 42 42 42 F F F 16 320-339 28 28 28 47 47 47 F F F 17 340-359 28 28 28 48 48 48 F F F 18 360-379 24 24 24 42 42 42 F F F 19 380-399 28 28 28 48 48 48 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 30 30 30 51 51 51 F F F 22 440-459 29 29 29 50 50 50 F F F 23 460-479 21 30 30 34 52 52 T T T <--- 24 480-499 30 30 30 51 53 53 F T T <--- 25 500-519 30 30 30 51 51 51 F F F 26 520-539 30 30 30 51 51 51 F T T <--- 27 540-559 23 23 23 36 36 36 F F F 28 560-579 24 24 24 39 39 39 F F F 29 580-599 25 25 25 43 43 43 F F F Max gen: 30, Max flush: 54 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 48 48 48 F F F 1 020-039 28 28 28 47 47 47 F F F 2 040-059 28 28 28 47 47 47 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 30 30 30 52 52 52 F F F 5 100-119 31 31 31 55 55 55 F F F 6 120-139 31 31 31 55 55 55 F F F 7 140-159 30 30 30 54 54 54 F F F 8 160-179 21 21 21 34 34 34 F F F 9 180-199 31 31 31 55 55 55 F F F 10 200-219 24 24 24 39 39 39 F F F 11 220-239 31 31 31 55 55 55 F F F 12 240-259 31 31 31 55 55 55 F F F 13 260-279 31 31 31 55 55 55 F F F 14 280-299 29 29 29 50 50 50 F F F 15 300-319 24 24 24 42 42 42 F F F 16 320-339 28 28 28 47 47 47 F F F 17 340-359 28 28 28 48 48 48 F F F 18 360-379 24 24 24 42 42 42 F F F 19 380-399 28 28 28 48 48 48 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 30 30 30 51 51 51 F F F 22 440-459 29 29 29 50 50 50 F F F 23 460-479 31 31 31 55 55 55 F F F 24 480-499 31 31 31 55 55 55 F F F 25 500-519 30 30 30 51 51 51 F F F 26 520-539 31 31 31 55 55 55 F F F 27 540-559 23 23 23 36 36 36 F F F 28 560-579 24 24 24 39 39 39 F F F 29 580-599 25 25 25 43 43 43 F F F Max gen: 31, Max flush: 55 Begin loop 16 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.036436894Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.08135407Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.082161079Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.082209475Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.082226124Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.082253991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328} {"msg":"Crucible de6e08fe-6c35-422d-a342-59a376ee965a has session id: 479cb001-fadf-400f-9341-ccc08c77db23","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.082737119Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Upstairs opts: Upstairs UUID: de6e08fe-6c35-422d-a342-59a376ee965a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.082877047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Crucible stats registered with UUID: de6e08fe-6c35-422d-a342-59a376ee965a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.082906025Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:01.082923993Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083131845Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083164332Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.08318297Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083424679Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"task":"crutest"} {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083456386Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328} {"msg":"de6e08fe-6c35-422d-a342-59a376ee965a active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083521731Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083541629Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"client":"0","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083559138Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"client":"1","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083575886Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"client":"2","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083592435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083615183Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083633251Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"ds_connection connected from Ok(127.0.0.1:46388)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083750581Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"ds_connection connected from Ok(127.0.0.1:49540)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083795367Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"ds_connection connected from Ok(127.0.0.1:33818)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.083829024Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.085081775Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"client":"1","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.085217793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"client":"0","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.085719999Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.085818551Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.086328316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"client":"2","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"[0]R flush_numbers[0..12]: [48, 47, 47, 55, 52, 55, 55, 54, 34, 55, 39, 55]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.086831312Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"[0]R generation[0..12]: [28, 28, 28, 31, 30, 31, 31, 30, 21, 31, 24, 31]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.08686145Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.086882368Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"[1]R flush_numbers[0..12]: [48, 47, 47, 55, 52, 55, 55, 54, 34, 55, 39, 55]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.086901656Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"[1]R generation[0..12]: [28, 28, 28, 31, 30, 31, 31, 30, 21, 31, 24, 31]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.086924134Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.086944513Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"[2]R flush_numbers[0..12]: [48, 47, 47, 55, 52, 55, 55, 54, 34, 55, 39, 55]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.086964231Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"[2]R generation[0..12]: [28, 28, 28, 31, 30, 31, 31, 30, 21, 31, 24, 31]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.086982639Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087002617Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087019516Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087036634Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Next flush: 56","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2025-12-24T00:17:01.087058823Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087075621Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"":"downstairs","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.08709288Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087108548Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"client":"0","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087125577Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"client":"1","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087142585Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"client":"2","":"downstairs","session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087159774Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087176093Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"de6e08fe-6c35-422d-a342-59a376ee965a is now active with session: 479cb001-fadf-400f-9341-ccc08c77db23","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087194161Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087215899Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328,"session_id":"479cb001-fadf-400f-9341-ccc08c77db23"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087232068Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328} {"msg":"Activated sub_volume de6e08fe-6c35-422d-a342-59a376ee965a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:01.087246186Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 591 len 12288 data: 3 2 2 02/30 Read block 176 len 4096 03/30 Flush 04/30 Write block 299 len 36864 data: 4 4 5 6 5 5 4 3 3 05/30 Flush 06/30 Write block 156 len 24576 data: 3 4 4 4 4 5 07/30 Write block 33 len 4096 data: 3 08/30 Read block 304 len 32768 09/30 Write block 215 len 32768 data: 4 4 4 4 4 5 5 6 10/30 Read block 400 len 40960 11/30 Read block 431 len 36864 12/30 Read block 192 len 24576 13/30 Read block 111 len 4096 14/30 Read block 279 len 20480 15/30 Read block 550 len 20480 16/30 Read block 62 len 4096 17/30 Read block 568 len 32768 18/30 Read block 228 len 32768 19/30 Read block 242 len 28672 20/30 Write block 105 len 12288 data: 4 4 4 21/30 Write block 459 len 12288 data: 4 2 4 22/30 Read block 175 len 24576 23/30 Read block 366 len 40960 24/30 Read block 304 len 12288 25/30 Read block 418 len 16384 26/30 Flush 27/30 Write block 556 len 20480 data: 3 3 2 2 2 28/30 Read block 4 len 12288 29/30 Read block 447 len 4096 30/30 Write block 21 len 20480 data: 5 4 6 6 6 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 9 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Read 8 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Read 9 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Read 5 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Write 3 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 27 29 Done 26 26 0 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) Some(JobId(1002)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:01.107433777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1328} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 48 48 48 F F F 1 020-039 32 32 28 58 58 47 T T F <--- 2 040-059 28 28 28 47 47 47 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 30 30 30 52 52 52 F F F 5 100-119 32 32 31 58 58 55 F F F <--- 6 120-139 31 31 31 55 55 55 F F F 7 140-159 32 32 30 58 58 54 F F F <--- 8 160-179 32 32 21 58 58 34 F F F <--- 9 180-199 31 31 31 55 55 55 F F F 10 200-219 32 32 24 58 58 39 F F F <--- 11 220-239 32 32 31 58 58 55 F F F <--- 12 240-259 31 31 31 55 55 55 F F F 13 260-279 31 31 31 55 55 55 F F F 14 280-299 32 32 29 57 57 50 F F F <--- 15 300-319 32 32 24 57 57 42 F F F <--- 16 320-339 28 28 28 47 47 47 F F F 17 340-359 28 28 28 48 48 48 F F F 18 360-379 24 24 24 42 42 42 F F F 19 380-399 28 28 28 48 48 48 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 30 30 30 51 51 51 F F F 22 440-459 32 32 29 58 58 50 F F F <--- 23 460-479 32 32 31 58 58 55 F F F <--- 24 480-499 31 31 31 55 55 55 F F F 25 500-519 30 30 30 51 51 51 F F F 26 520-539 31 31 31 55 55 55 F F F 27 540-559 23 23 23 36 36 36 T T F <--- 28 560-579 24 24 24 39 39 39 T T F <--- 29 580-599 32 32 32 56 56 56 F F F Max gen: 32, Max flush: 58 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 48 48 48 F F F 1 020-039 33 33 33 59 59 59 F F F 2 040-059 28 28 28 47 47 47 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 30 30 30 52 52 52 F F F 5 100-119 32 32 32 58 58 58 F F F 6 120-139 31 31 31 55 55 55 F F F 7 140-159 32 32 32 58 58 58 F F F 8 160-179 32 32 32 58 58 58 F F F 9 180-199 31 31 31 55 55 55 F F F 10 200-219 32 32 32 58 58 58 F F F 11 220-239 32 32 32 58 58 58 F F F 12 240-259 31 31 31 55 55 55 F F F 13 260-279 31 31 31 55 55 55 F F F 14 280-299 32 32 32 57 57 57 F F F 15 300-319 32 32 32 57 57 57 F F F 16 320-339 28 28 28 47 47 47 F F F 17 340-359 28 28 28 48 48 48 F F F 18 360-379 24 24 24 42 42 42 F F F 19 380-399 28 28 28 48 48 48 F F F 20 400-419 21 21 21 34 34 34 F F F 21 420-439 30 30 30 51 51 51 F F F 22 440-459 32 32 32 58 58 58 F F F 23 460-479 32 32 32 58 58 58 F F F 24 480-499 31 31 31 55 55 55 F F F 25 500-519 30 30 30 51 51 51 F F F 26 520-539 31 31 31 55 55 55 F F F 27 540-559 33 33 33 59 59 59 F F F 28 560-579 33 33 33 59 59 59 F F F 29 580-599 32 32 32 56 56 56 F F F Max gen: 33, Max flush: 59 Begin loop 17 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.508988352Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.554487747Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.555224673Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.555273218Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.555289957Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.555311755Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336} {"msg":"Crucible 984ef4c8-dbcb-421f-b237-21501bfb4abe has session id: c9955486-3433-4c76-b30e-7723bfdc7d3f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.555866197Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Upstairs opts: Upstairs UUID: 984ef4c8-dbcb-421f-b237-21501bfb4abe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.555924352Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Crucible stats registered with UUID: 984ef4c8-dbcb-421f-b237-21501bfb4abe","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.555954059Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:05.555980297Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556232735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556279921Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556300309Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556481973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"task":"crutest"} {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556521989Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336} {"msg":"984ef4c8-dbcb-421f-b237-21501bfb4abe active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556556186Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556592503Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"client":"0","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.5566313Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"client":"1","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556664507Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"client":"2","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556698994Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556734311Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556767588Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"ds_connection connected from Ok(127.0.0.1:57570)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556802595Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"ds_connection connected from Ok(127.0.0.1:33659)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556891937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"ds_connection connected from Ok(127.0.0.1:50850)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.556938053Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558056756Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"client":"2","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558102832Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"client":"0","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558142848Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"client":"1","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558662893Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558760774Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"[0]R flush_numbers[0..12]: [48, 59, 47, 55, 52, 58, 55, 58, 58, 55, 58, 58]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558847597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"[0]R generation[0..12]: [28, 33, 28, 31, 30, 32, 31, 32, 32, 31, 32, 32]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558873635Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558894483Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"[1]R flush_numbers[0..12]: [48, 59, 47, 55, 52, 58, 55, 58, 58, 55, 58, 58]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558913621Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"[1]R generation[0..12]: [28, 33, 28, 31, 30, 32, 31, 32, 32, 31, 32, 32]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558932479Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558952398Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"[2]R flush_numbers[0..12]: [48, 59, 47, 55, 52, 58, 55, 58, 58, 55, 58, 58]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558971026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"[2]R generation[0..12]: [28, 33, 28, 31, 30, 32, 31, 32, 32, 31, 32, 32]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.558989485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559018882Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Max found gen is 34","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:17:05.559048639Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559071247Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Next flush: 60","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559088626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559105334Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"":"downstairs","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559125353Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559140841Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"client":"0","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.55915805Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"client":"1","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559175188Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"client":"2","":"downstairs","session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559192337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559208535Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"984ef4c8-dbcb-421f-b237-21501bfb4abe is now active with session: c9955486-3433-4c76-b30e-7723bfdc7d3f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559226434Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559241743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336,"session_id":"c9955486-3433-4c76-b30e-7723bfdc7d3f"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.559257911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336} {"msg":"Activated sub_volume 984ef4c8-dbcb-421f-b237-21501bfb4abe","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:05.55927187Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 272 len 12288 data: 6 5 4 02/30 Read block 520 len 8192 03/30 Write block 482 len 12288 data: 4 4 5 04/30 Read block 179 len 28672 05/30 Write block 578 len 12288 data: 4 3 4 06/30 Read block 286 len 32768 07/30 Write block 467 len 24576 data: 5 6 5 6 6 5 08/30 Write block 337 len 16384 data: 4 3 2 3 09/30 Read block 117 len 40960 10/30 Flush 11/30 Write block 272 len 16384 data: 7 6 5 4 12/30 Write block 478 len 28672 data: 2 2 3 4 5 5 6 13/30 Read block 495 len 20480 14/30 Write block 102 len 16384 data: 4 5 4 5 15/30 Read block 528 len 32768 16/30 Read block 159 len 16384 17/30 Read block 464 len 40960 18/30 Write block 388 len 4096 data: 4 19/30 Read block 344 len 8192 20/30 Read block 21 len 12288 21/30 Write block 451 len 20480 data: 5 4 4 4 4 22/30 Read block 306 len 28672 23/30 Flush 24/30 Write block 5 len 40960 data: 4 3 2 2 2 3 2 2 2 2 25/30 Write block 396 len 24576 data: 3 4 3 3 3 2 26/30 Write block 469 len 8192 data: 6 7 27/30 Read block 53 len 20480 28/30 Write block 45 len 24576 data: 4 3 2 2 2 3 29/30 Write block 204 len 40960 data: 3 2 2 2 2 2 2 2 4 4 30/30 Read block 84 len 40960 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 2 Done Sent Done false 1002 Acked Write 3 Done Sent Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Read 3 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Write 10 Done Sent Done false 1029 Acked Read 10 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:05.578734094Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1336} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 28 28 28 48 48 48 T F T <--- 1 020-039 33 33 33 59 59 59 F F F 2 040-059 28 28 28 47 47 47 T F T <--- 3 060-079 31 31 31 55 55 55 F F F 4 080-099 30 30 30 52 52 52 F F F 5 100-119 34 32 34 61 58 61 F F F <--- 6 120-139 31 31 31 55 55 55 F F F 7 140-159 32 32 32 58 58 58 F F F 8 160-179 32 32 32 58 58 58 F F F 9 180-199 31 31 31 55 55 55 F F F 10 200-219 32 32 32 58 58 58 T F T <--- 11 220-239 32 32 32 58 58 58 F F F 12 240-259 31 31 31 55 55 55 F F F 13 260-279 34 31 34 61 55 61 F T F <--- 14 280-299 32 32 32 57 57 57 F F F 15 300-319 32 32 32 57 57 57 F F F 16 320-339 34 28 34 60 47 60 F F F <--- 17 340-359 34 28 34 60 48 60 F F F <--- 18 360-379 24 24 24 42 42 42 F F F 19 380-399 34 28 34 61 48 61 T F T <--- 20 400-419 21 21 21 34 34 34 T F T <--- 21 420-439 30 30 30 51 51 51 F F F 22 440-459 34 32 34 61 58 61 F F F <--- 23 460-479 34 32 34 61 58 61 T F T <--- 24 480-499 34 31 34 61 55 61 F F F <--- 25 500-519 30 30 30 51 51 51 F F F 26 520-539 31 31 31 55 55 55 F F F 27 540-559 33 33 33 59 59 59 F F F 28 560-579 34 33 34 60 59 60 F F F <--- 29 580-599 34 32 34 60 56 60 F F F <--- Max gen: 34, Max flush: 61 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 62 62 62 F F F 1 020-039 33 33 33 59 59 59 F F F 2 040-059 35 35 35 62 62 62 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 30 30 30 52 52 52 F F F 5 100-119 34 34 34 61 61 61 F F F 6 120-139 31 31 31 55 55 55 F F F 7 140-159 32 32 32 58 58 58 F F F 8 160-179 32 32 32 58 58 58 F F F 9 180-199 31 31 31 55 55 55 F F F 10 200-219 35 35 35 62 62 62 F F F 11 220-239 32 32 32 58 58 58 F F F 12 240-259 31 31 31 55 55 55 F F F 13 260-279 34 34 34 61 61 61 F F F 14 280-299 32 32 32 57 57 57 F F F 15 300-319 32 32 32 57 57 57 F F F 16 320-339 34 34 34 60 60 60 F F F 17 340-359 34 34 34 60 60 60 F F F 18 360-379 24 24 24 42 42 42 F F F 19 380-399 35 35 35 62 62 62 F F F 20 400-419 35 35 35 62 62 62 F F F 21 420-439 30 30 30 51 51 51 F F F 22 440-459 34 34 34 61 61 61 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 34 34 34 61 61 61 F F F 25 500-519 30 30 30 51 51 51 F F F 26 520-539 31 31 31 55 55 55 F F F 27 540-559 33 33 33 59 59 59 F F F 28 560-579 34 34 34 60 60 60 F F F 29 580-599 34 34 34 60 60 60 F F F Max gen: 35, Max flush: 62 Begin loop 18 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:09.991060859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:10.03442571Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.034825785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.065810985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.066470588Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.066526863Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.066550321Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.066579918Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344} {"msg":"Crucible 8535f110-10f3-449c-a903-4b7566c312ff has session id: be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067178136Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Upstairs opts: Upstairs UUID: 8535f110-10f3-449c-a903-4b7566c312ff, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067225162Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Crucible stats registered with UUID: 8535f110-10f3-449c-a903-4b7566c312ff","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.0672495Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:10.067270958Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067442633Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067488179Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067522906Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067652735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"task":"crutest"} {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067673273Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344} {"msg":"8535f110-10f3-449c-a903-4b7566c312ff active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067739677Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067759875Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"client":"0","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067777694Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"client":"1","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067794632Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"client":"2","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067811491Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067832819Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067850788Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"ds_connection connected from Ok(127.0.0.1:61959)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.0679432Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"ds_connection connected from Ok(127.0.0.1:51298)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.067991025Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"ds_connection connected from Ok(127.0.0.1:40035)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.06805413Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.06930911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"client":"1","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.0694254Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"client":"0","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.069470066Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"client":"2","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.069910768Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.069983052Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"[0]R flush_numbers[0..12]: [62, 59, 62, 55, 52, 61, 55, 58, 58, 55, 62, 58]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070069824Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"[0]R generation[0..12]: [35, 33, 35, 31, 30, 34, 31, 32, 32, 31, 35, 32]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070095422Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.07011626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"[1]R flush_numbers[0..12]: [62, 59, 62, 55, 52, 61, 55, 58, 58, 55, 62, 58]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070135368Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"[1]R generation[0..12]: [35, 33, 35, 31, 30, 34, 31, 32, 32, 31, 35, 32]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070158446Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070178685Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"[2]R flush_numbers[0..12]: [62, 59, 62, 55, 52, 61, 55, 58, 58, 55, 62, 58]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070197183Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"[2]R generation[0..12]: [35, 33, 35, 31, 30, 34, 31, 32, 32, 31, 35, 32]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070215601Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.07023521Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070251528Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070275366Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Next flush: 63","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070291615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070307943Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"":"downstairs","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070325242Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.07034092Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"client":"0","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070357949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"client":"1","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070375117Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"client":"2","":"downstairs","session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070392476Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070408694Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"8535f110-10f3-449c-a903-4b7566c312ff is now active with session: be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070428203Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"Set Active after no reconciliation","v":0,Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "name":"crucible","level":30,"time":"2025-12-24T00:17:10.070447371Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344,"session_id":"be1ab373-e8c7-4be0-a7c5-3a0a0bb9b026"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.07046379Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344} {"msg":"Activated sub_volume 8535f110-10f3-449c-a903-4b7566c312ff","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:10.070477788Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 530 len 16384 data: 4 4 4 4 02/30 Write block 138 len 8192 data: 2 2 03/30 Write block 521 len 8192 data: 2 3 04/30 Flush 05/30 Write block 347 len 16384 data: 4 4 4 4 06/30 Write block 492 len 20480 data: 3 3 3 5 5 07/30 Read block 389 len 8192 08/30 Flush 09/30 Write block 305 len 12288 data: 5 4 4 10/30 Read block 124 len 8192 11/30 Write block 51 len 20480 data: 2 2 2 2 2 12/30 Flush 13/30 Write block 235 len 28672 data: 3 4 3 3 3 3 3 14/30 Write block 577 len 8192 data: 2 5 15/30 Read block 112 len 36864 16/30 Write block 480 len 4096 data: 4 17/30 Write block 2 len 36864 data: 2 2 2 5 4 3 3 3 4 18/30 Read block 565 len 20480 19/30 Flush 20/30 Write block 384 len 32768 data: 4 4 3 4 5 3 2 2 21/30 Write block 231 len 12288 data: 3 3 3 22/30 Read block 359 len 8192 23/30 Read block 500 len 40960 24/30 Write block 594 len 8192 data: 2 2 25/30 Write block 524 len 40960 data: 2 3 2 2 2 2 5 5 5 5 26/30 Read block 107 len 12288 27/30 Read block 549 len 28672 28/30 Write block 306 len 32768 data: 5 5 3 2 2 2 2 2 29/30 Read block 223 len 36864 30/30 Write block 106 len 4096 data: 5 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 4 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 3 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Write 2 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Read 10 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Read 7 Sent Done Done false 1027 Acked Write 8 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 1 1 28 Done 0 25 25 50 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:10.088217593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1344} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 36 36 62 66 66 F F F <--- 1 020-039 33 33 33 59 59 59 F F F 2 040-059 35 36 36 62 65 65 F F F <--- 3 060-079 31 31 31 55 55 55 F F F 4 080-099 30 30 30 52 52 52 F F F 5 100-119 34 34 34 61 61 61 F T T <--- 6 120-139 36 36 36 63 63 63 F F F 7 140-159 32 32 32 58 58 58 F F F 8 160-179 32 32 32 58 58 58 F F F 9 180-199 31 31 31 55 55 55 F F F 10 200-219 35 35 35 62 62 62 F F F 11 220-239 32 36 36 58 66 66 F T T <--- 12 240-259 31 36 36 55 66 66 F F F <--- 13 260-279 34 34 34 61 61 61 F F F 14 280-299 32 32 32 57 57 57 F F F 15 300-319 32 36 36 57 65 65 F T T <--- 16 320-339 34 34 34 60 60 60 F F F 17 340-359 34 36 36 60 64 64 F F F <--- 18 360-379 24 24 24 42 42 42 F F F 19 380-399 35 35 35 62 62 62 F T T <--- 20 400-419 35 35 35 62 62 62 F F F 21 420-439 30 30 30 51 51 51 F F F 22 440-459 34 34 34 61 61 61 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 34 36 36 61 66 66 F F F <--- 25 500-519 30 30 30 51 51 51 F F F 26 520-539 36 36 36 63 63 63 F T T <--- 27 540-559 33 33 33 59 59 59 F F F 28 560-579 34 36 36 60 66 66 F F F <--- 29 580-599 34 34 34 60 60 60 F T T <--- Max gen: 36, Max flush: 66 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 66 66 66 F F F 1 020-039 33 33 33 59 59 59 F F F 2 040-059 36 36 36 65 65 65 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 30 30 30 52 52 52 F F F 5 100-119 37 37 37 67 67 67 F F F 6 120-139 36 36 36 63 63 63 F F F 7 140-159 32 32 32 58 58 58 F F F 8 160-179 32 32 32 58 58 58 F F F 9 180-199 31 31 31 55 55 55 F F F 10 200-219 35 35 35 62 62 62 F F F 11 220-239 37 37 37 67 67 67 F F F 12 240-259 36 36 36 66 66 66 F F F 13 260-279 34 34 34 61 61 61 F F F 14 280-299 32 32 32 57 57 57 F F F 15 300-319 37 37 37 67 67 67 F F F 16 320-339 34 34 34 60 60 60 F F F 17 340-359 36 36 36 64 64 64 F F F 18 360-379 24 24 24 42 42 42 F F F 19 380-399 37 37 37 67 67 67 F F F 20 400-419 35 35 35 62 62 62 F F F 21 420-439 30 30 30 51 51 51 F F F 22 440-459 34 34 34 61 61 61 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 36 36 36 66 66 66 F F F 25 500-519 30 30 30 51 51 51 F F F 26 520-539 37 37 37 67 67 67 F F F 27 540-559 33 33 33 59 59 59 F F F 28 560-579 36 36 36 66 66 66 F F F 29 580-599 37 37 37 67 67 67 F F F Max gen: 37, Max flush: 67 Begin loop 19 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.527055338Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.57212879Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.572830609Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.572890614Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.572911322Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.57293882Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352} {"msg":"Crucible fce519c6-dad3-4912-a353-bfe097da6792 has session id: e6cf7ae4-b5b9-4d11-95bd-58aad8113035","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.57339191Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Upstairs opts: Upstairs UUID: fce519c6-dad3-4912-a353-bfe097da6792, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.573503381Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Crucible stats registered with UUID: fce519c6-dad3-4912-a353-bfe097da6792","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.573539917Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:14.573566965Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.573802434Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.573839431Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.573861579Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574003047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"task":"crutest"} {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574031854Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352} {"msg":"fce519c6-dad3-4912-a353-bfe097da6792 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574048253Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574064342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"client":"0","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.57408194Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"client":"1","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574098979Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"client":"2","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574115647Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574133296Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574150614Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"ds_connection connected from Ok(127.0.0.1:52893)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574403132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"ds_connection connected from Ok(127.0.0.1:34190)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574447018Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"ds_connection connected from Ok(127.0.0.1:44731)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.574529381Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.575498757Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"client":"0","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.575652573Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"client":"2","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576092575Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576203395Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"client":"1","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576232253Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"[0]R flush_numbers[0..12]: [66, 59, 65, 55, 52, 67, 63, 58, 58, 55, 62, 67]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576783105Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"[0]R generation[0..12]: [36, 33, 36, 31, 30, 37, 36, 32, 32, 31, 35, 37]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576818152Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.57684374Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"[1]R flush_numbers[0..12]: [66, 59, 65, 55, 52, 67, 63, 58, 58, 55, 62, 67]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576867158Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"[1]R generation[0..12]: [36, 33, 36, 31, 30, 37, 36, 32, 32, 31, 35, 37]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576890436Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576915233Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"[2]R flush_numbers[0..12]: [66, 59, 65, 55, 52, 67, 63, 58, 58, 55, 62, 67]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576938331Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"[2]R generation[0..12]: [36, 33, 36, 31, 30, 37, 36, 32, 32, 31, 35, 37]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576961009Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.576985187Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Max found gen is 38","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577005466Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577026384Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Next flush: 68","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577046262Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"All extents match","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2025-12-24T00:17:14.577077629Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"":"downstairs","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577099827Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577119056Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"client":"0","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577140144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"client":"1","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577161122Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"client":"2","":"downstairs","session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.57718223Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577202848Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"fce519c6-dad3-4912-a353-bfe097da6792 is now active with session: e6cf7ae4-b5b9-4d11-95bd-58aad8113035","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577225576Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577244685Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352,"session_id":"e6cf7ae4-b5b9-4d11-95bd-58aad8113035"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577264523Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352} {"msg":"Activated sub_volume fce519c6-dad3-4912-a353-bfe097da6792","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:14.577284101Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 348 len 12288 data: 5 5 5 02/30 Read block 315 len 40960 03/30 Write block 450 len 36864 data: 7 6 5 5 5 5 4 4 4 04/30 Write block 319 len 20480 data: 2 2 2 2 2 05/30 Write block 315 len 20480 data: 2 2 2 2 3 06/30 Read block 413 len 24576 07/30 Read block 100 len 20480 08/30 Read block 531 len 12288 09/30 Read block 457 len 8192 10/30 Write block 418 len 20480 data: 2 2 2 3 3 11/30 Read block 18 len 40960 12/30 Write block 339 len 8192 data: 3 4 13/30 Read block 12 len 8192 14/30 Write block 82 len 16384 data: 6 8 6 4 15/30 Read block 218 len 12288 16/30 Write block 414 len 28672 data: 4 4 4 2 3 3 3 17/30 Read block 119 len 40960 18/30 Flush 19/30 Write block 359 len 36864 data: 5 5 5 4 4 5 4 4 4 20/30 Read block 372 len 32768 21/30 Read block 98 len 32768 22/30 Read block 132 len 4096 23/30 Write block 127 len 28672 data: 5 5 5 3 3 3 4 24/30 Flush 25/30 Flush 26/30 Write block 524 len 4096 data: 3 27/30 Write block 395 len 36864 data: 3 4 5 4 4 4 3 3 4 28/30 Read block 325 len 28672 29/30 Read block 567 len 40960 30/30 Write block 201 len 24576 data: 6 4 4 4 3 3 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Write 9 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Read 6 Done Done Done false 1006 Acked Read 5 Done Done Done false 1007 Acked Read 3 Done Done Done false 1008 Acked Read 2 Done Done Done false 1009 Acked Write 5 Done Done Done false 1010 Acked Read 10 Done Done Done false 1011 Acked Write 2 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 19 1 21 Done 29 11 29 69 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:14.597368151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1352} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 66 66 66 F F F 1 020-039 33 33 33 59 59 59 F F F 2 040-059 36 36 36 65 65 65 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 38 30 38 68 52 68 F F F <--- 5 100-119 37 37 37 67 67 67 F F F 6 120-139 38 36 38 69 63 69 F F F <--- 7 140-159 32 32 32 58 58 58 F F F 8 160-179 32 32 32 58 58 58 F F F 9 180-199 31 31 31 55 55 55 F F F 10 200-219 35 35 35 62 62 62 T F T <--- 11 220-239 37 37 37 67 67 67 F F F 12 240-259 36 36 36 66 66 66 F F F 13 260-279 34 34 34 61 61 61 F F F 14 280-299 32 32 32 57 57 57 F F F 15 300-319 38 37 38 68 67 68 F T F <--- 16 320-339 38 34 38 68 60 68 F T F <--- 17 340-359 38 36 38 69 64 69 F T F <--- 18 360-379 38 24 38 69 42 69 F F F <--- 19 380-399 37 37 37 67 67 67 T F T <--- 20 400-419 38 35 38 68 62 68 T T T <--- 21 420-439 38 30 38 68 51 68 F T F <--- 22 440-459 38 34 38 68 61 68 F T F <--- 23 460-479 35 35 35 62 62 62 F F F 24 480-499 36 36 36 66 66 66 F F F 25 500-519 30 30 30 51 51 51 F F F 26 520-539 37 37 37 67 67 67 T F T <--- 27 540-559 33 33 33 59 59 59 F F F 28 560-579 36 36 36 66 66 66 F F F 29 580-599 37 37 37 67 67 67 F F F Max gen: 38, Max flush: 69 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 66 66 66 F F F 1 020-039 33 33 33 59 59 59 F F F 2 040-059 36 36 36 65 65 65 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 38 38 38 68 68 68 F F F 5 100-119 37 37 37 67 67 67 F F F 6 120-139 38 38 38 69 69 69 F F F 7 140-159 32 32 32 58 58 58 F F F 8 160-179 32 32 32 58 58 58 F F F 9 180-199 31 31 31 55 55 55 F F F 10 200-219 39 39 39 70 70 70 F F F 11 220-239 37 37 37 67 67 67 F F F 12 240-259 36 36 36 66 66 66 F F F 13 260-279 34 34 34 61 61 61 F F F 14 280-299 32 32 32 57 57 57 F F F 15 300-319 38 38 38 68 68 68 F F F 16 320-339 38 38 38 68 68 68 F F F 17 340-359 38 38 38 69 69 69 F F F 18 360-379 38 38 38 69 69 69 F F F 19 380-399 39 39 39 70 70 70 F F F 20 400-419 39 39 39 70 70 70 F F F 21 420-439 38 38 38 68 68 68 F F F 22 440-459 38 38 38 68 68 68 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 36 36 36 66 66 66 F F F 25 500-519 30 30 30 51 51 51 F F F 26 520-539 39 39 39 70 70 70 F F F 27 540-559 33 33 33 59 59 59 F F F 28 560-579 36 36 36 66 66 66 F F F 29 580-599 37 37 37 67 67 67 F F F Max gen: 39, Max flush: 70 Begin loop 20 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.002798218Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.049696381Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.050441786Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.050497421Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.050519959Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.050542917Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360} {"msg":"Crucible 2ec19f65-7487-44cf-9300-7b4b64d65d43 has session id: a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051043313Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Upstairs opts: Upstairs UUID: 2ec19f65-7487-44cf-9300-7b4b64d65d43, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051100088Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Crucible stats registered with UUID: 2ec19f65-7487-44cf-9300-7b4b64d65d43","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051130846Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:19.051156983Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051374635Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051413551Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051433279Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051608754Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"task":"crutest"} {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051637462Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360} {"msg":"2ec19f65-7487-44cf-9300-7b4b64d65d43 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051706706Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051726484Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"client":"0","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051756521Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"client":"1","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051788499Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"client":"2","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051822416Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051856853Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051894429Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"ds_connection connected from Ok(127.0.0.1:58289)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051926307Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"ds_connection connected from Ok(127.0.0.1:52074)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051959324Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"ds_connection connected from Ok(127.0.0.1:57108)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.051991191Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.053074776Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"client":"0","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.053183977Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"client":"2","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.053582442Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.053737429Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.053806153Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"client":"1","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"[0]R flush_numbers[0..12]: [66, 59, 65, 55, 68, 67, 69, 58, 58, 55, 70, 67]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054330367Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"[0]R generation[0..12]: [36, 33, 36, 31, 38, 37, 38, 32, 32, 31, 39, 37]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054362754Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054383902Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"[1]R flush_numbers[0..12]: [66, 59, 65, 55, 68, 67, 69, 58, 58, 55, 70, 67]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054403081Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"[1]R generation[0..12]: [36, 33, 36, 31, 38, 37, 38, 32, 32, 31, 39, 37]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054422069Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054441937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"[2]R flush_numbers[0..12]: [66, 59, 65, 55, 68, 67, 69, 58, 58, 55, 70, 67]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054465025Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"[2]R generation[0..12]: [36, 33, 36, 31, 38, 37, 38, 32, 32, 31, 39, 37]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054484264Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054504132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:17:19.054520701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054544448Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Next flush: 71","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054560887Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054577316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"":"downstairs","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054594384Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054609903Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"client":"0","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054627111Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"client":"1","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054652169Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"client":"2","":"downstairs","session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:17:19.054681276Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054699405Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"2ec19f65-7487-44cf-9300-7b4b64d65d43 is now active with session: a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054717743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054735852Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360,"session_id":"a4fd5c84-7ecc-401a-ac7d-ce1ab78effd5"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.05475221Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360} {"msg":"Activated sub_volume 2ec19f65-7487-44cf-9300-7b4b64d65d43","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:19.054766609Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 281 len 4096 data: 3 02/30 Write block 442 len 40960 data: 2 3 4 6 7 8 6 6 8 7 03/30 Read block 360 len 12288 04/30 Read block 282 len 24576 05/30 Write block 274 len 4096 data: 6 06/30 Read block 365 len 20480 07/30 Write block 137 len 24576 data: 2 3 3 2 3 3 08/30 Flush 09/30 Write block 273 len 4096 data: 7 10/30 Write block 540 len 8192 data: 4 4 11/30 Write block 150 len 12288 data: 3 4 3 12/30 Write block 26 len 24576 data: 6 6 4 4 3 3 13/30 Write block 503 len 4096 data: 3 14/30 Write block 207 len 4096 data: 3 15/30 Read block 497 len 24576 16/30 Read block 153 len 24576 17/30 Read block 406 len 16384 18/30 Flush 19/30 Write block 553 len 12288 data: 4 3 3 20/30 Flush 21/30 Write block 529 len 4096 data: 3 22/30 Write block 564 len 12288 data: 2 2 2 23/30 Read block 463 len 12288 24/30 Read block 464 len 12288 25/30 Read block 310 len 12288 26/30 Read block 302 len 4096 27/30 Write block 510 len 20480 data: 3 3 3 3 3 28/30 Read block 242 len 24576 29/30 Read block 517 len 12288 30/30 Write block 10 len 40960 data: 5 3 3 3 3 3 3 3 4 5 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Write 1 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 1 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:19.071618791Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1360} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 66 66 66 T F T <--- 1 020-039 40 33 40 72 59 72 F F F <--- 2 040-059 36 36 36 65 65 65 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 38 38 38 68 68 68 F F F 5 100-119 37 37 37 67 67 67 F F F 6 120-139 40 38 40 71 69 71 F F F <--- 7 140-159 40 32 40 72 58 72 F F F <--- 8 160-179 32 32 32 58 58 58 F F F 9 180-199 31 31 31 55 55 55 F F F 10 200-219 40 39 40 72 70 72 F F F <--- 11 220-239 37 37 37 67 67 67 F F F 12 240-259 36 36 36 66 66 66 F F F 13 260-279 40 34 40 72 61 72 F F F <--- 14 280-299 40 32 40 71 57 71 F T F <--- 15 300-319 38 38 38 68 68 68 F F F 16 320-339 38 38 38 68 68 68 F F F 17 340-359 38 38 38 69 69 69 F F F 18 360-379 38 38 38 69 69 69 F F F 19 380-399 39 39 39 70 70 70 F F F 20 400-419 39 39 39 70 70 70 F F F 21 420-439 38 38 38 68 68 68 F F F 22 440-459 40 38 40 71 68 71 F T F <--- 23 460-479 35 35 35 62 62 62 F F F 24 480-499 36 36 36 66 66 66 F F F 25 500-519 40 30 40 72 51 72 T F T <--- 26 520-539 39 39 39 70 70 70 T F T <--- 27 540-559 40 33 40 73 59 73 F F F <--- 28 560-579 36 36 36 66 66 66 T F T <--- 29 580-599 37 37 37 67 67 67 F F F Max gen: 40, Max flush: 73 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 41 41 41 74 74 74 F F F 1 020-039 40 40 40 72 72 72 F F F 2 040-059 36 36 36 65 65 65 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 38 38 38 68 68 68 F F F 5 100-119 37 37 37 67 67 67 F F F 6 120-139 40 40 40 71 71 71 F F F 7 140-159 40 40 40 72 72 72 F F F 8 160-179 32 32 32 58 58 58 F F F 9 180-199 31 31 31 55 55 55 F F F 10 200-219 40 40 40 72 72 72 F F F 11 220-239 37 37 37 67 67 67 F F F 12 240-259 36 36 36 66 66 66 F F F 13 260-279 40 40 40 72 72 72 F F F 14 280-299 40 40 40 71 71 71 F F F 15 300-319 38 38 38 68 68 68 F F F 16 320-339 38 38 38 68 68 68 F F F 17 340-359 38 38 38 69 69 69 F F F 18 360-379 38 38 38 69 69 69 F F F 19 380-399 39 39 39 70 70 70 F F F 20 400-419 39 39 39 70 70 70 F F F 21 420-439 38 38 38 68 68 68 F F F 22 440-459 40 40 40 71 71 71 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 36 36 36 66 66 66 F F F 25 500-519 41 41 41 74 74 74 F F F 26 520-539 41 41 41 74 74 74 F F F 27 540-559 40 40 40 73 73 73 F F F 28 560-579 41 41 41 74 74 74 F F F 29 580-599 37 37 37 67 67 67 F F F Max gen: 41, Max flush: 74 Begin loop 21 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.481632728Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.526644735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.527360833Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.527408749Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.527425307Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.527447395Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368} {"msg":"Crucible 85c5bd1c-f8b4-4ab1-9df9-e70c83378b57 has session id: f18ead2e-d0a3-4a26-b7e6-085111200ef9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.52796486Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Upstairs opts: Upstairs UUID: 85c5bd1c-f8b4-4ab1-9df9-e70c83378b57, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528028025Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Crucible stats registered with UUID: 85c5bd1c-f8b4-4ab1-9df9-e70c83378b57","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528062952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:23.528091379Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528276053Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.52831326Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528331958Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528581346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"task":"crutest"} {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528601735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368} {"msg":"85c5bd1c-f8b4-4ab1-9df9-e70c83378b57 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528674868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528695027Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"client":"0","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528716895Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"client":"1","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528746532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"client":"2","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528778779Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528812256Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528844963Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"ds_connection connected from Ok(127.0.0.1:48644)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.52887928Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"ds_connection connected from Ok(127.0.0.1:59413)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528911278Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"ds_connection connected from Ok(127.0.0.1:59057)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.528944445Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.530102304Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"client":"2","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.530209265Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"client":"0","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.530653176Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.530736109Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.531538659Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"client":"1","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"[0]R flush_numbers[0..12]: [74, 72, 65, 55, 68, 67, 71, 72, 58, 55, 72, 67]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532029516Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"[0]R generation[0..12]: [41, 40, 36, 31, 38, 37, 40, 40, 32, 31, 40, 37]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532058003Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532078872Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"[1]R flush_numbers[0..12]: [74, 72, 65, 55, 68, 67, 71, 72, 58, 55, 72, 67]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.53209795Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"[1]R generation[0..12]: [41, 40, 36, 31, 38, 37, 40, 40, 32, 31, 40, 37]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532116748Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532136437Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"[2]R flush_numbers[0..12]: [74, 72, 65, 55, 68, 67, 71, 72, 58, 55, 72, 67]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532154995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"[2]R generation[0..12]: [41, 40, 36, 31, 38, 37, 40, 40, 32, 31, 40, 37]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532173333Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532195291Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO time":"2025-12-24T00:17:23.532222929Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532261766Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Next flush: 75","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532309361Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532345289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"":"downstairs","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532384465Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532417672Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"client":"0","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2025-12-24T00:17:23.532451169Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"client":"1","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532485316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"client":"2","":"downstairs","session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532505055Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532521563Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"85c5bd1c-f8b4-4ab1-9df9-e70c83378b57 is now active with session: f18ead2e-d0a3-4a26-b7e6-085111200ef9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532539532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.53255523Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368,"session_id":"f18ead2e-d0a3-4a26-b7e6-085111200ef9"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532571409Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368} {"msg":"Activated sub_volume 85c5bd1c-f8b4-4ab1-9df9-e70c83378b57","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:23.532585408Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 196 len 40960 data: 3 4 7 7 5 7 5 5 5 4 02/30 Write block 164 len 12288 data: 4 4 3 03/30 Read block 140 len 4096 04/30 Read block 519 len 36864 05/30 Flush 06/30 Write block 170 len 32768 data: 3 3 3 4 4 4 5 4 07/30 Read block 260 len 36864 08/30 Write block 579 len 16384 data: 4 5 4 3 09/30 Read block 16 len 32768 10/30 Read block 531 len 12288 11/30 Write block 570 len 4096 data: 4 12/30 Write block 259 len 24576 data: 3 3 3 3 3 2 13/30 Read block 21 len 24576 14/30 Write block 141 len 20480 data: 4 4 3 3 4 15/30 Write block 238 len 40960 data: 4 4 4 4 4 4 3 2 4 5 16/30 Write block 258 len 28672 data: 3 4 4 4 4 4 3 17/30 Write block 87 len 36864 data: 4 2 2 2 2 3 3 4 4 18/30 Read block 428 len 40960 19/30 Read block 575 len 28672 20/30 Write block 321 len 36864 data: 3 3 3 4 4 4 4 4 4 21/30 Write block 36 len 32768 data: 2 2 2 4 5 4 5 5 22/30 Write block 255 len 40960 data: 5 5 6 4 5 5 5 5 5 4 23/30 Read block 232 len 40960 24/30 Write block 396 len 16384 data: 5 6 5 5 25/30 Flush 26/30 Write block 280 len 4096 data: 3 27/30 Write block 12 len 20480 data: 4 4 4 4 4 28/30 Read block 558 len 28672 29/30 Read block 317 len 12288 30/30 Write block 117 len 36864 data: 4 4 3 3 5 4 4 4 4 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 8 Done Done Done false 1006 Acked Read 9 Done Sent Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Read 10 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 24 1 26 Done 24 1 24 49 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1004)) Some(JobId(1024)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:23.552344616Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1368} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 41 41 41 74 74 74 T F T <--- 1 020-039 42 40 42 76 72 76 F F F <--- 2 040-059 42 36 42 76 65 76 F F F <--- 3 060-079 31 31 31 55 55 55 F F F 4 080-099 42 38 42 76 68 76 F F F <--- 5 100-119 37 37 37 67 67 67 T F T <--- 6 120-139 40 40 40 71 71 71 T F T <--- 7 140-159 42 40 42 76 72 76 F F F <--- 8 160-179 42 42 42 76 75 76 F T F <--- 9 180-199 42 42 42 75 75 75 F F F 10 200-219 42 42 42 75 75 75 F F F 11 220-239 42 37 42 76 67 76 F F F <--- 12 240-259 42 36 42 76 66 76 F F F <--- 13 260-279 42 40 42 76 72 76 F F F <--- 14 280-299 40 40 40 71 71 71 T F T <--- 15 300-319 38 38 38 68 68 68 F F F 16 320-339 42 38 42 76 68 76 F F F <--- 17 340-359 38 38 38 69 69 69 F F F 18 360-379 38 38 38 69 69 69 F F F 19 380-399 42 39 42 76 70 76 F F F <--- 20 400-419 39 39 39 70 70 70 F F F 21 420-439 38 38 38 68 68 68 F F F 22 440-459 40 40 40 71 71 71 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 36 36 36 66 66 66 F F F 25 500-519 41 41 41 74 74 74 F F F 26 520-539 41 41 41 74 74 74 F F F 27 540-559 40 40 40 73 73 73 F F F 28 560-579 42 41 42 76 74 76 F F F <--- 29 580-599 42 37 42 76 67 76 F F F <--- Max gen: 42, Max flush: 76 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 43 43 43 77 77 77 F F F 1 020-039 42 42 42 76 76 76 F F F 2 040-059 42 42 42 76 76 76 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 42 42 42 76 76 76 F F F 5 100-119 43 43 43 77 77 77 F F F 6 120-139 43 43 43 77 77 77 F F F 7 140-159 42 42 42 76 76 76 F F F 8 160-179 42 42 42 76 76 76 F F F 9 180-199 42 42 42 75 75 75 F F F 10 200-219 42 42 42 75 75 75 F F F 11 220-239 42 42 42 76 76 76 F F F 12 240-259 42 42 42 76 76 76 F F F 13 260-279 42 42 42 76 76 76 F F F 14 280-299 43 43 43 77 77 77 F F F 15 300-319 38 38 38 68 68 68 F F F 16 320-339 42 42 42 76 76 76 F F F 17 340-359 38 38 38 69 69 69 F F F 18 360-379 38 38 38 69 69 69 F F F 19 380-399 42 42 42 76 76 76 F F F 20 400-419 39 39 39 70 70 70 F F F 21 420-439 38 38 38 68 68 68 F F F 22 440-459 40 40 40 71 71 71 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 36 36 36 66 66 66 F F F 25 500-519 41 41 41 74 74 74 F F F 26 520-539 41 41 41 74 74 74 F F F 27 540-559 40 40 40 73 73 73 F F F 28 560-579 42 42 42 76 76 76 F F F 29 580-599 42 42 42 76 76 76 F F F Max gen: 43, Max flush: 77 Begin loop 22 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:27.971885881Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.016734093Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.017502157Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.017550122Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.017567061Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.017589069Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376} {"msg":"Crucible ccbef0cd-5fb8-420c-8830-ad3c035d5988 has session id: e51e870d-04ba-4e7d-ba65-9375b8d1fc31","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018098525Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Upstairs opts: Upstairs UUID: ccbef0cd-5fb8-420c-8830-ad3c035d5988, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018158929Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Crucible stats registered with UUID: ccbef0cd-5fb8-420c-8830-ad3c035d5988","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018189886Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:28.018217954Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018478501Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018518408Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018538636Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018744328Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"task":"crutest"} {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018771066Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376} {"msg":"ccbef0cd-5fb8-420c-8830-ad3c035d5988 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.01883361Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018853879Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"client":"0","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018871547Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"client":"1","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018888406Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"client":"2","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018905054Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018928962Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018959639Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"ds_connection connected from Ok(127.0.0.1:58062)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.018997076Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"ds_connection connected from Ok(127.0.0.1:56300)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.019030453Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"ds_connection connected from Ok(127.0.0.1:56082)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.019126295Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.020241188Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"client":"0","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.020382905Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"client":"2","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.020774201Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.020933987Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.02136196Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"client":"1","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"[0]R flush_numbers[0..12]: [77, 76, 76, 55, 76, 77, 77, 76, 76, 75, 75, 76]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.021849718Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"[0]R generation[0..12]: [43, 42, 42, 31, 42, 43, 43, 42, 42, 42, 42, 42]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.021874556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.021895474Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"[1]R flush_numbers[0..12]: [77, 76, 76, 55, 76, 77, 77, 76, 76, 75, 75, 76]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.021919581Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"[1]R generation[0..12]: [43, 42, 42, 31, 42, 43, 43, 42, 42, 42, 42, 42]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.02194251Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.021967127Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"[2]R flush_numbers[0..12]: [77, 76, 76, 55, 76, 77, 77, 76, 76, 75, 75, 76]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.021993615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"[2]R generation[0..12]: [43, 42, 42, 31, 42, 43, 43, 42, 42, 42, 42, 42]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022017453Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022041931Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Max found gen is 44","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022062479Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022083677Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Next flush: 78","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022103705Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022131043Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"":"downstairs","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022152281Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.02217159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"client":"0","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022192748Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"client":"1","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022214386Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"client":"2","":"downstairs","session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022235984Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2025-12-24T00:17:28.022255992Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"ccbef0cd-5fb8-420c-8830-ad3c035d5988 is now active with session: e51e870d-04ba-4e7d-ba65-9375b8d1fc31","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.02228122Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022300578Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376,"session_id":"e51e870d-04ba-4e7d-ba65-9375b8d1fc31"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022320886Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376} {"msg":"Activated sub_volume ccbef0cd-5fb8-420c-8830-ad3c035d5988","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:28.022340285Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 585 len 4096 data: 3 02/30 Write block 290 len 8192 data: 4 4 03/30 Flush 04/30 Write block 98 len 16384 data: 5 5 6 5 05/30 Write block 354 len 8192 data: 5 6 06/30 Read block 120 len 8192 07/30 Read block 217 len 32768 08/30 Read block 132 len 24576 09/30 Write block 290 len 12288 data: 5 5 5 10/30 Write block 26 len 32768 data: 7 7 5 5 4 4 3 4 11/30 Write block 91 len 24576 data: 3 4 4 5 5 4 12/30 Read block 372 len 36864 13/30 Read block 245 len 12288 14/30 Read block 592 len 32768 15/30 Read block 131 len 20480 16/30 Read block 281 len 28672 17/30 Write block 525 len 16384 data: 4 3 3 3 18/30 Read block 143 len 16384 19/30 Write block 333 len 32768 data: 3 4 5 4 5 4 4 5 20/30 Read block 461 len 20480 21/30 Read block 393 len 12288 22/30 Write block 131 len 20480 data: 4 4 5 4 5 23/30 Write block 482 len 24576 data: 6 6 7 4 5 5 24/30 Read block 129 len 40960 25/30 Write block 114 len 4096 data: 4 26/30 Flush 27/30 Write block 499 len 24576 data: 5 5 5 3 4 3 28/30 Read block 402 len 16384 29/30 Read block 118 len 16384 30/30 Read block 588 len 40960 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 2 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Write 2 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 5 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Write 5 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Read 10 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:28.041461878Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1376} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 43 43 43 77 77 77 F F F 1 020-039 44 42 44 79 76 79 F F F <--- 2 040-059 42 42 42 76 76 76 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 44 42 44 79 76 79 F F F <--- 5 100-119 44 43 44 79 77 79 F F F <--- 6 120-139 44 43 44 79 77 79 F F F <--- 7 140-159 42 42 42 76 76 76 F F F 8 160-179 42 42 42 76 76 76 F F F 9 180-199 42 42 42 75 75 75 F F F 10 200-219 42 42 42 75 75 75 F F F 11 220-239 42 42 42 76 76 76 F F F 12 240-259 42 42 42 76 76 76 F F F 13 260-279 42 42 42 76 76 76 F F F 14 280-299 44 43 44 79 77 79 F F F <--- 15 300-319 38 38 38 68 68 68 F F F 16 320-339 44 42 44 79 76 79 F F F <--- 17 340-359 44 38 44 79 69 79 F F F <--- 18 360-379 38 38 38 69 69 69 F F F 19 380-399 42 42 42 76 76 76 F F F 20 400-419 39 39 39 70 70 70 F F F 21 420-439 38 38 38 68 68 68 F F F 22 440-459 40 40 40 71 71 71 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 44 36 44 79 66 79 T F T <--- 25 500-519 41 41 41 74 74 74 T F T <--- 26 520-539 44 41 44 79 74 79 F F F <--- 27 540-559 40 40 40 73 73 73 F F F 28 560-579 42 42 42 76 76 76 F F F 29 580-599 44 42 44 78 76 78 F T F <--- Max gen: 44, Max flush: 79 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 22 Verifying data now No change after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 43 43 43 77 77 77 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 42 42 42 76 76 76 F F F 3 060-079 31 31 31 55 55 55 F F F 4 080-099 44 44 44 79 79 79 F F F 5 100-119 44 44 44 79 79 79 F F F 6 120-139 44 44 44 79 79 79 F F F 7 140-159 42 42 42 76 76 76 F F F 8 160-179 42 42 42 76 76 76 F F F 9 180-199 42 42 42 75 75 75 F F F 10 200-219 42 42 42 75 75 75 F F F 11 220-239 42 42 42 76 76 76 F F F 12 240-259 42 42 42 76 76 76 F F F 13 260-279 42 42 42 76 76 76 F F F 14 280-299 44 44 44 79 79 79 F F F 15 300-319 38 38 38 68 68 68 F F F 16 320-339 44 44 44 79 79 79 F F F 17 340-359 44 44 44 79 79 79 F F F 18 360-379 38 38 38 69 69 69 F F F 19 380-399 42 42 42 76 76 76 F F F 20 400-419 39 39 39 70 70 70 F F F 21 420-439 38 38 38 68 68 68 F F F 22 440-459 40 40 40 71 71 71 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 45 45 45 80 80 80 F F F 25 500-519 45 45 45 80 80 80 F F F 26 520-539 44 44 44 79 79 79 F F F 27 540-559 40 40 40 73 73 73 F F F 28 560-579 42 42 42 76 76 76 F F F 29 580-599 44 44 44 78 78 78 F F F Max gen: 45, Max flush: 80 Begin loop 23 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.46141129Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.506200317Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.50707671Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.507132905Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.507153034Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.507180071Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384} {"msg":"Crucible 9fc30d8e-9281-48b9-b2a8-69ffffc5751e has session id: 878548db-8062-41b7-9b40-bce0547cc0b1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.507808487Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Upstairs opts: Upstairs UUID: 9fc30d8e-9281-48b9-b2a8-69ffffc5751e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.5078788Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Crucible stats registered with UUID: 9fc30d8e-9281-48b9-b2a8-69ffffc5751e","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.507913388Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:32.507934866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508160616Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508209112Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508239649Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508368208Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"task":"crutest"} {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508392916Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384} {"msg":"9fc30d8e-9281-48b9-b2a8-69ffffc5751e active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508476448Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508497406Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"client":"0","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508523444Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"client":"1","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508555381Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"client":"2","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508587439Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508632395Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508668972Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"ds_connection connected from Ok(127.0.0.1:60961)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508703389Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"ds_connection connected from Ok(127.0.0.1:33067)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.508738216Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"ds_connection connected from Ok(127.0.0.1:61514)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.50879899Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.510024653Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"client":"1","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.510168161Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"client":"0","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.51063927Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.510762109Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511251077Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"client":"2","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"[0]R flush_numbers[0..12]: [77, 79, 76, 55, 79, 79, 79, 76, 76, 75, 75, 76]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511758682Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"[0]R generation[0..12]: [43, 44, 42, 31, 44, 44, 44, 42, 42, 42, 42, 42]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.51178816Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511809738Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"[1]R flush_numbers[0..12]: [77, 79, 76, 55, 79, 79, 79, 76, 76, 75, 75, 76]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511829076Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"[1]R generation[0..12]: [43, 44, 42, 31, 44, 44, 44, 42, 42, 42, 42, 42]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511847755Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511867433Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"[2]R flush_numbers[0..12]: [77, 79, 76, 55, 79, 79, 79, 76, 76, 75, 75, 76]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511886171Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"[2]R generation[0..12]: [43, 44, 42, 31, 44, 44, 44, 42, 42, 42, 42, 42]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.51190461Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511923988Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511940337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511957285Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Next flush: 81","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.511973404Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"Wait for a query_work_queue command to finish before sending IO downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.51200964Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"":"downstairs","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.512029429Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.512049417Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"client":"0","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.512066655Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"client":"1","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.512083494Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"client":"2","":"downstairs","session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.512100393Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.512116551Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"9fc30d8e-9281-48b9-b2a8-69ffffc5751e is now active with session: 878548db-8062-41b7-9b40-bce0547cc0b1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.51213439Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.512149928Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384,"session_id":"878548db-8062-41b7-9b40-bce0547cc0b1"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.512166047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384} {"msg":"Activated sub_volume 9fc30d8e-9281-48b9-b2a8-69ffffc5751e","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:32.512180386Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 274 len 12288 data: 7 5 5 02/30 Read block 226 len 40960 03/30 Read block 468 len 8192 04/30 Read block 321 len 12288 05/30 Write block 256 len 20480 data: 6 7 5 6 6 06/30 Write block 155 len 36864 data: 3 4 5 5 5 5 6 4 4 07/30 Read block 590 len 28672 08/30 Write block 505 len 40960 data: 4 4 3 3 5 4 4 4 4 4 09/30 Read block 47 len 32768 10/30 Write block 536 len 12288 data: 5 5 5 11/30 Read block 411 len 32768 12/30 Write block 381 len 24576 data: 2 2 2 5 5 4 13/30 Read block 178 len 16384 14/30 Write block 18 len 8192 data: 5 6 15/30 Read block 336 len 28672 16/30 Read block 564 len 8192 17/30 Flush 18/30 Write block 375 len 40960 data: 3 2 2 2 2 2 3 3 3 6 19/30 Write block 218 len 16384 data: 5 5 6 6 20/30 Read block 481 len 24576 21/30 Read block 73 len 28672 22/30 Read block 359 len 28672 23/30 Write block 284 len 20480 data: 3 3 3 3 3 24/30 Read block 475 len 32768 25/30 Read block 555 len 24576 26/30 Read block 447 len 36864 27/30 Write block 321 len 12288 data: 4 4 4 28/30 Write block 419 len 36864 data: 4 4 4 4 4 4 4 5 5 29/30 Write block 53 len 40960 data: 3 3 3 2 2 4 4 4 3 3 30/30 Read block 217 len 24576 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Sent false 1001 Acked Read 10 Done Done Sent false 1002 Acked Read 2 Done Done Sent false 1003 Acked Read 3 Done Done Sent false 1004 Acked Write 5 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Read 7 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Write 10 Done Done Sent false 1029 Acked Read 6 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 30 31 Done 29 30 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:32.532578498Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1384} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 46 46 43 81 81 77 F F F <--- 1 020-039 44 44 44 79 79 79 F F F 2 040-059 42 42 42 76 76 76 T T F <--- 3 060-079 31 31 31 55 55 55 T T F <--- 4 080-099 44 44 44 79 79 79 F F F 5 100-119 44 44 44 79 79 79 F F F 6 120-139 44 44 44 79 79 79 F F F 7 140-159 46 46 42 81 81 76 F F F <--- 8 160-179 46 46 42 81 81 76 F F F <--- 9 180-199 42 42 42 75 75 75 F F F 10 200-219 42 42 42 75 75 75 T T F <--- 11 220-239 42 42 42 76 76 76 T T F <--- 12 240-259 46 46 42 81 81 76 F F F <--- 13 260-279 46 46 42 81 81 76 F F F <--- 14 280-299 44 44 44 79 79 79 T T F <--- 15 300-319 38 38 38 68 68 68 F F F 16 320-339 44 44 44 79 79 79 T T F <--- 17 340-359 44 44 44 79 79 79 F F F 18 360-379 38 38 38 69 69 69 T T F <--- 19 380-399 46 46 42 81 81 76 T T F <--- 20 400-419 39 39 39 70 70 70 T T F <--- 21 420-439 38 38 38 68 68 68 T T F <--- 22 440-459 40 40 40 71 71 71 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 45 45 45 80 80 80 F F F 25 500-519 46 46 45 81 81 80 F F F <--- 26 520-539 46 46 44 81 81 79 F F F <--- 27 540-559 40 40 40 73 73 73 F F F 28 560-579 42 42 42 76 76 76 F F F 29 580-599 44 44 44 78 78 78 F F F Max gen: 46, Max flush: 81 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 46 46 46 81 81 81 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 47 47 47 82 82 82 F F F 4 080-099 44 44 44 79 79 79 F F F 5 100-119 44 44 44 79 79 79 F F F 6 120-139 44 44 44 79 79 79 F F F 7 140-159 46 46 46 81 81 81 F F F 8 160-179 46 46 46 81 81 81 F F F 9 180-199 42 42 42 75 75 75 F F F 10 200-219 47 47 47 82 82 82 F F F 11 220-239 47 47 47 82 82 82 F F F 12 240-259 46 46 46 81 81 81 F F F 13 260-279 46 46 46 81 81 81 F F F 14 280-299 47 47 47 82 82 82 F F F 15 300-319 38 38 38 68 68 68 F F F 16 320-339 47 47 47 82 82 82 F F F 17 340-359 44 44 44 79 79 79 F F F 18 360-379 47 47 47 82 82 82 F F F 19 380-399 47 47 47 82 82 82 F F F 20 400-419 47 47 47 82 82 82 F F F 21 420-439 47 47 47 82 82 82 F F F 22 440-459 40 40 40 71 71 71 F F F 23 460-479 35 35 35 62 62 62 F F F 24 480-499 45 45 45 80 80 80 F F F 25 500-519 46 46 46 81 81 81 F F F 26 520-539 46 46 46 81 81 81 F F F 27 540-559 40 40 40 73 73 73 F F F 28 560-579 42 42 42 76 76 76 F F F 29 580-599 44 44 44 78 78 78 F F F Max gen: 47, Max flush: 82 Begin loop 24 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:36.966515379Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:37.009751161Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.010183913Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.040948512Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.041617744Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.04166401Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.041680119Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.041702447Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392} {"msg":"Crucible 15350ba2-c265-428a-b197-73a4fe0928e1 has session id: 95b5dee2-8d81-4aa7-8cd4-dc1923831e3b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042164117Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Upstairs opts: Upstairs UUID: 15350ba2-c265-428a-b197-73a4fe0928e1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042209882Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Crucible stats registered with UUID: 15350ba2-c265-428a-b197-73a4fe0928e1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042230281Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:37.042247159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042458211Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042500697Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042520506Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042675942Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"task":"crutest"} {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.04269853Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392} {"msg":"15350ba2-c265-428a-b197-73a4fe0928e1 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042768514Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042790072Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"client":"0","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.04281886Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"client":"1","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042850747Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"client":"2","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042885684Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042919151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.042959557Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"ds_connection connected from Ok(127.0.0.1:59675)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.04315925Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"ds_connection connected from Ok(127.0.0.1:41744)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.043198686Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"ds_connection connected from Ok(127.0.0.1:48184)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.043220525Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.044317159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"client":"1","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.0444234Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"client":"0","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.044476715Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"client":"2","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.044883209Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045008079Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"[0]R flush_numbers[0..12]: [81, 79, 82, 82, 79, 79, 79, 81, 81, 75, 82, 82]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045046305Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"[0]R generation[0..12]: [46, 44, 47, 47, 44, 44, 44, 46, 46, 42, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045066344Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045086412Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"[1]R flush_numbers[0..12]: [81, 79, 82, 82, 79, 79, 79, 81, 81, 75, 82, 82]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.04510526Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"[1]R generation[0..12]: [46, 44, 47, 47, 44, 44, 44, 46, 46, 42, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045124048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045147006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"[2]R flush_numbers[0..12]: [81, 79, 82, 82, 79, 79, 79, 81, 81, 75, 82, 82]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045166105Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"[2]R generation[0..12]: [46, 44, 47, 47, 44, 44, 44, 46, 46, 42, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045184503Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045204112Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045228579Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:17:37.045258687Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Next flush: 83","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045293364Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045313632Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"":"downstairs","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.04533105Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045346569Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"client":"0","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045363828Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"client":"1","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045380926Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"client":"2","":"downstairs","session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045398365Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045414793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"15350ba2-c265-428a-b197-73a4fe0928e1 is now active with session: 95b5dee2-8d81-4aa7-8cd4-dc1923831e3b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045432982Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.04544833Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392,"session_id":"95b5dee2-8d81-4aa7-8cd4-dc1923831e3b"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045466969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392} {"msg":"Activated sub_volume 15350ba2-c265-428a-b197-73a4fe0928e1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:37.045481368Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 376 len 36864 data: 3 3 3 3 3 4 4 4 7 02/30 Flush 03/30 Write block 583 len 12288 data: 3 3 4 04/30 Read block 324 len 24576 05/30 Read block 180 len 36864 06/30 Write block 176 len 24576 data: 6 5 4 4 5 5 07/30 Flush 08/30 Write block 435 len 12288 data: 3 3 2 09/30 Write block 189 len 4096 data: 5 10/30 Write block 138 len 32768 data: 4 4 3 5 5 4 4 5 11/30 Read block 18 len 40960 12/30 Read block 7 len 4096 13/30 Write block 471 len 20480 data: 7 6 5 5 5 14/30 Write block 384 len 28672 data: 8 6 5 5 6 4 3 15/30 Read block 330 len 4096 16/30 Write block 122 len 8192 data: 5 5 17/30 Write block 477 len 4096 data: 2 18/30 Write block 76 len 8192 data: 7 7 19/30 Write block 331 len 12288 data: 4 4 4 20/30 Read block 578 len 12288 21/30 Read block 179 len 8192 22/30 Flush 23/30 Write block 262 len 4096 data: 6 24/30 Write block 449 len 8192 data: 7 9 25/30 Flush 26/30 Write block 123 len 36864 data: 6 5 5 4 6 6 6 4 5 27/30 Read block 64 len 4096 28/30 Read block 313 len 12288 29/30 Read block 320 len 40960 30/30 Read block 272 len 8192 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 3 Done Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Write 7 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 2 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 1 0 28 Done 1 27 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:37.063457961Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1392} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 46 46 46 81 81 81 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 47 48 48 82 85 85 F F F <--- 4 080-099 44 44 44 79 79 79 F F F 5 100-119 44 44 44 79 79 79 F F F 6 120-139 44 48 48 79 85 85 F T T <--- 7 140-159 46 48 48 81 85 85 F F F <--- 8 160-179 46 48 48 81 84 84 F F F <--- 9 180-199 42 48 48 75 85 85 F F F <--- 10 200-219 47 47 47 82 82 82 F F F 11 220-239 47 47 47 82 82 82 F F F 12 240-259 46 46 46 81 81 81 F F F 13 260-279 46 48 48 81 86 86 F F F <--- 14 280-299 47 47 47 82 82 82 F F F 15 300-319 38 38 38 68 68 68 F F F 16 320-339 47 48 48 82 85 85 F F F <--- 17 340-359 44 44 44 79 79 79 F F F 18 360-379 48 48 48 83 83 83 F F F 19 380-399 48 48 48 83 85 85 F F F <--- 20 400-419 47 47 47 82 82 82 F F F 21 420-439 47 48 48 82 85 85 F F F <--- 22 440-459 40 48 48 71 86 86 F F F <--- 23 460-479 35 48 48 62 85 85 F F F <--- 24 480-499 45 45 45 80 80 80 F F F 25 500-519 46 46 46 81 81 81 F F F 26 520-539 46 46 46 81 81 81 F F F 27 540-559 40 40 40 73 73 73 F F F 28 560-579 42 42 42 76 76 76 F F F 29 580-599 44 48 48 78 84 84 T F F <--- Max gen: 48, Max flush: 86 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 24 Verifying data now No change after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 46 46 46 81 81 81 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 48 48 48 85 85 85 F F F 4 080-099 44 44 44 79 79 79 F F F 5 100-119 44 44 44 79 79 79 F F F 6 120-139 49 49 49 87 87 87 F F F 7 140-159 48 48 48 85 85 85 F F F 8 160-179 48 48 48 84 84 84 F F F 9 180-199 48 48 48 85 85 85 F F F 10 200-219 47 47 47 82 82 82 F F F 11 220-239 47 47 47 82 82 82 F F F 12 240-259 46 46 46 81 81 81 F F F 13 260-279 48 48 48 86 86 86 F F F 14 280-299 47 47 47 82 82 82 F F F 15 300-319 38 38 38 68 68 68 F F F 16 320-339 48 48 48 85 85 85 F F F 17 340-359 44 44 44 79 79 79 F F F 18 360-379 48 48 48 83 83 83 F F F 19 380-399 48 48 48 85 85 85 F F F 20 400-419 47 47 47 82 82 82 F F F 21 420-439 48 48 48 85 85 85 F F F 22 440-459 48 48 48 86 86 86 F F F 23 460-479 48 48 48 85 85 85 F F F 24 480-499 45 45 45 80 80 80 F F F 25 500-519 46 46 46 81 81 81 F F F 26 520-539 46 46 46 81 81 81 F F F 27 540-559 40 40 40 73 73 73 F F F 28 560-579 42 42 42 76 76 76 F F F 29 580-599 48 48 48 84 84 84 F F F Max gen: 49, Max flush: 87 Begin loop 25 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.494531052Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.539974012Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.540724057Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.540772402Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.540790221Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.540812589Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400} {"msg":"Crucible 58952db7-778e-4003-a107-6ec65a28c6e7 has session id: 0585804d-024f-412c-9896-cafad3e4ade6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.541384659Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Upstairs opts: Upstairs UUID: 58952db7-778e-4003-a107-6ec65a28c6e7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.541441714Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Crucible stats registered with UUID: 58952db7-778e-4003-a107-6ec65a28c6e7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.541472312Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:41.541498799Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.541708561Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.541757187Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.541778695Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.541881786Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"task":"crutest"} {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.541903404Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400} {"msg":"58952db7-778e-4003-a107-6ec65a28c6e7 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.541961729Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.541992766Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"client":"0","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.542024633Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"client":"1","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.542054171Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"client":"2","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.542085528Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.542118825Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.542153942Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"ds_connection connected from Ok(127.0.0.1:53534)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.542190359Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"ds_connection connected from Ok(127.0.0.1:58619)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.542323147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"ds_connection connected from Ok(127.0.0.1:44326)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.542362254Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.543470147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"client":"0","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.543549471Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"client":"1","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.543594096Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"client":"2","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544060526Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544157107Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"[0]R flush_numbers[0..12]: [81, 79, 82, 85, 79, 79, 87, 85, 84, 85, 82, 82]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544232701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"[0]R generation[0..12]: [46, 44, 47, 48, 44, 44, 49, 48, 48, 48, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544257449Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544277827Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"[1]R flush_numbers[0..12]: [81, 79, 82, 85, 79, 79, 87, 85, 84, 85, 82, 82]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544296465Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"[1]R generation[0..12]: [46, 44, 47, 48, 44, 44, 49, 48, 48, 48, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544314934Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544334592Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"[2]R flush_numbers[0..12]: [81, 79, 82, 85, 79, 79, 87, 85, 84, 85, 82, 82]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544352901Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"[2]R generation[0..12]: [46, 44, 47, 48, 44, 44, 49, 48, 48, 48, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544371779Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544391447Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544407996Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544427804Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Next flush: 88"Wait for a query_work_queue command to finish before sending IO ,"v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544457991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.54447509Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"":"downstairs","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544495468Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544511317Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"client":"0","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544528265Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"client":"1","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544545184Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"client":"2","":"downstairs","session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544562412Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544578441Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"58952db7-778e-4003-a107-6ec65a28c6e7 is now active with session: 0585804d-024f-412c-9896-cafad3e4ade6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544596399Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544611668Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400,"session_id":"0585804d-024f-412c-9896-cafad3e4ade6"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544627626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400} {"msg":"Activated sub_volume 58952db7-778e-4003-a107-6ec65a28c6e7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:41.544641795Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 243 len 20480 data: 5 4 3 5 6 02/30 Read block 455 len 16384 03/30 Read block 116 len 16384 04/30 Read block 162 len 32768 05/30 Read block 252 len 28672 06/30 Read block 47 len 28672 07/30 Write block 374 len 24576 data: 3 4 4 4 4 4 08/30 Read block 435 len 8192 09/30 Write block 9 len 4096 data: 4 10/30 Read block 370 len 8192 11/30 Flush 12/30 Write block 171 len 40960 data: 4 4 5 5 5 7 6 5 5 6 13/30 Write block 314 len 20480 data: 2 3 3 3 3 14/30 Read block 507 len 20480 15/30 Write block 249 len 20480 data: 7 6 6 5 5 16/30 Read block 435 len 40960 17/30 Write block 542 len 28672 data: 4 3 2 2 3 3 4 18/30 Read block 204 len 16384 19/30 Read block 459 len 16384 20/30 Write block 82 len 20480 data: 7 9 7 5 4 21/30 Write block 474 len 8192 data: 6 6 22/30 Read block 362 len 20480 23/30 Write block 85 len 16384 data: 6 5 5 3 24/30 Read block 569 len 28672 25/30 Read block 412 len 4096 26/30 Write block 358 len 36864 data: 5 6 6 6 5 5 6 5 5 27/30 Read block 555 len 28672 28/30 Write block 160 len 12288 data: 6 7 5 29/30 Write block 161 len 16384 data: 8 6 5 5 30/30 Read block 445 len 16384 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 4 Done Done Sent false 1002 Acked Read 4 Done Done Sent false 1003 Acked Read 8 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Read 2 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Write 5 Done Done Sent false 1015 Acked Read 10 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 4 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Write 5 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Read 5 Done Done Sent false 1022 Acked Write 4 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Read 7 Done Done Sent false 1027 Acked Write 3 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:41.565075105Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1400} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 46 88 88 81 F F F <--- 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 48 48 48 85 85 85 F F F 4 080-099 44 44 44 79 79 79 T T F <--- 5 100-119 44 44 44 79 79 79 F F F 6 120-139 49 49 49 87 87 87 F F F 7 140-159 48 48 48 85 85 85 F F F 8 160-179 48 48 48 84 84 84 T T F <--- 9 180-199 48 48 48 85 85 85 T T F <--- 10 200-219 47 47 47 82 82 82 F F F 11 220-239 47 47 47 82 82 82 F F F 12 240-259 50 50 46 88 88 81 T T T <--- 13 260-279 48 48 48 86 86 86 F F F 14 280-299 47 47 47 82 82 82 F F F 15 300-319 38 38 38 68 68 68 T T F <--- 16 320-339 48 48 48 85 85 85 F F F 17 340-359 44 44 44 79 79 79 T T F <--- 18 360-379 50 50 48 88 88 83 T T F <--- 19 380-399 48 48 48 85 85 85 F F F 20 400-419 47 47 47 82 82 82 F F F 21 420-439 48 48 48 85 85 85 F F F 22 440-459 48 48 48 86 86 86 F F F 23 460-479 48 48 48 85 85 85 T T F <--- 24 480-499 45 45 45 80 80 80 F F F 25 500-519 46 46 46 81 81 81 F F F 26 520-539 46 46 46 81 81 81 F F F 27 540-559 40 40 40 73 73 73 T T F <--- 28 560-579 42 42 42 76 76 76 F F F 29 580-599 48 48 48 84 84 84 F F F Max gen: 50, Max flush: 88 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 25 Verifying data now No change after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 88 88 88 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 48 48 48 85 85 85 F F F 4 080-099 51 51 51 89 89 89 F F F 5 100-119 44 44 44 79 79 79 F F F 6 120-139 49 49 49 87 87 87 F F F 7 140-159 48 48 48 85 85 85 F F F 8 160-179 51 51 51 89 89 89 F F F 9 180-199 51 51 51 89 89 89 F F F 10 200-219 47 47 47 82 82 82 F F F 11 220-239 47 47 47 82 82 82 F F F 12 240-259 51 51 51 89 89 89 F F F 13 260-279 48 48 48 86 86 86 F F F 14 280-299 47 47 47 82 82 82 F F F 15 300-319 51 51 51 89 89 89 F F F 16 320-339 48 48 48 85 85 85 F F F 17 340-359 51 51 51 89 89 89 F F F 18 360-379 51 51 51 89 89 89 F F F 19 380-399 48 48 48 85 85 85 F F F 20 400-419 47 47 47 82 82 82 F F F 21 420-439 48 48 48 85 85 85 F F F 22 440-459 48 48 48 86 86 86 F F F 23 460-479 51 51 51 89 89 89 F F F 24 480-499 45 45 45 80 80 80 F F F 25 500-519 46 46 46 81 81 81 F F F 26 520-539 46 46 46 81 81 81 F F F 27 540-559 51 51 51 89 89 89 F F F 28 560-579 42 42 42 76 76 76 F F F 29 580-599 48 48 48 84 84 84 F F F Max gen: 51, Max flush: 89 Begin loop 26 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:45.967600105Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.012387592Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.013132367Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.013188872Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.013218299Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.013264935Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408} {"msg":"Crucible e5603588-b1f5-42c3-b5aa-febce71b0393 has session id: 7904fba6-c282-42ef-80a9-90baea826539","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.013680949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Upstairs opts: Upstairs UUID: e5603588-b1f5-42c3-b5aa-febce71b0393, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.013777911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Crucible stats registered with UUID: e5603588-b1f5-42c3-b5aa-febce71b0393","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.013815987Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:46.013843435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014034398Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014077815Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014110592Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014277717Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"task":"crutest"} {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014305515Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408} {"msg":"e5603588-b1f5-42c3-b5aa-febce71b0393 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014373679Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014394297Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"client":"0","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014411955Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"client":"1","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014428944Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"client":"2","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014445443Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014537835Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014580131Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"ds_connection connected from Ok(127.0.0.1:45410)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014617737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"ds_connection connected from Ok(127.0.0.1:36427)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014650165Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"ds_connection connected from Ok(127.0.0.1:34835)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.014683412Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.015889457Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"client":"0","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.015951731Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"client":"1","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.016478195Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.016510483Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.016585866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"client":"2","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"[0]R flush_numbers[0..12]: [88, 79, 82, 85, 89, 79, 87, 85, 89, 89, 82, 82]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017102411Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"[0]R generation[0..12]: [50, 44, 47, 48, 51, 44, 49, 48, 51, 51, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017128079Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017148407Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"[1]R flush_numbers[0..12]: [88, 79, 82, 85, 89, 79, 87, 85, 89, 89, 82, 82]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017167115Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"[1]R generation[0..12]: [50, 44, 47, 48, 51, 44, 49, 48, 51, 51, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017185584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017205272Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"[2]R flush_numbers[0..12]: [88, 79, 82, 85, 89, 79, 87, 85, 89, 89, 82, 82]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.01722697Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"[2]R generation[0..12]: [50, 44, 47, 48, 51, 44, 49, 48, 51, 51, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017245908Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017265687Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017282215Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017299224Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Next flush: 90","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017315412Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017331781Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"":"downstairs","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:17:46.017348799Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017371508Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"client":"0","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017388396Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"client":"1","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017405205Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"client":"2","":"downstairs","session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017422503Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017438722Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"e5603588-b1f5-42c3-b5aa-febce71b0393 is now active with session: 7904fba6-c282-42ef-80a9-90baea826539","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017467609Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017485858Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408,"session_id":"7904fba6-c282-42ef-80a9-90baea826539"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017502106Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408} {"msg":"Activated sub_volume e5603588-b1f5-42c3-b5aa-febce71b0393","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:46.017518385Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 161 len 12288 data: 9 7 6 02/30 Write block 190 len 24576 data: 3 3 2 2 2 2 03/30 Read block 152 len 4096 04/30 Write block 227 len 32768 data: 4 4 3 3 4 4 4 3 05/30 Read block 25 len 16384 06/30 Write block 310 len 12288 data: 3 3 3 07/30 Read block 436 len 32768 08/30 Write block 427 len 4096 data: 6 09/30 Read block 298 len 24576 10/30 Read block 321 len 32768 11/30 Write block 77 len 24576 data: 8 6 5 7 6 8 12/30 Read block 220 len 36864 13/30 Read block 22 len 36864 14/30 Write block 430 len 20480 data: 3 4 5 6 4 15/30 Write block 195 len 16384 data: 3 4 5 8 16/30 Read block 341 len 32768 17/30 Read block 135 len 36864 18/30 Read block 461 len 32768 19/30 Write block 274 len 36864 data: 8 6 6 6 3 3 4 4 3 20/30 Write block 411 len 16384 data: 5 3 4 5 21/30 Read block 232 len 4096 22/30 Write block 303 len 4096 data: 6 23/30 Read block 342 len 4096 24/30 Flush 25/30 Write block 264 len 16384 data: 5 2 2 3 26/30 Write block 541 len 24576 data: 5 5 4 3 3 4 27/30 Write block 285 len 16384 data: 4 4 4 4 28/30 Read block 481 len 32768 29/30 Write block 577 len 16384 data: 3 6 5 6 30/30 Write block 241 len 28672 data: 5 5 6 5 4 6 7 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 8 Done Done Sent false 1004 Acked Read 4 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Write 1 Done Done Sent false 1008 Acked Read 6 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Write 5 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Read 8 Done Done Sent false 1016 Acked Read 9 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 27 31 Done 28 28 3 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:46.035770384Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1408} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 88 88 88 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 52 52 48 90 90 85 F F F <--- 4 080-099 52 52 51 90 90 89 F F F <--- 5 100-119 44 44 44 79 79 79 F F F 6 120-139 49 49 49 87 87 87 F F F 7 140-159 48 48 48 85 85 85 F F F 8 160-179 52 52 51 90 90 89 F F T <--- 9 180-199 52 52 51 90 90 89 F F T <--- 10 200-219 47 47 47 82 82 82 F F F 11 220-239 52 52 47 90 90 82 F F F <--- 12 240-259 51 51 51 89 89 89 T T F <--- 13 260-279 52 52 48 90 90 86 T T F <--- 14 280-299 52 52 47 90 90 82 T T F <--- 15 300-319 52 52 51 90 90 89 F F F <--- 16 320-339 48 48 48 85 85 85 F F F 17 340-359 51 51 51 89 89 89 F F F 18 360-379 51 51 51 89 89 89 F F F 19 380-399 48 48 48 85 85 85 F F F 20 400-419 52 52 47 90 90 82 F F F <--- 21 420-439 52 52 48 90 90 85 F F F <--- 22 440-459 48 48 48 86 86 86 F F F 23 460-479 51 51 51 89 89 89 F F F 24 480-499 45 45 45 80 80 80 F F F 25 500-519 46 46 46 81 81 81 F F F 26 520-539 46 46 46 81 81 81 F F F 27 540-559 51 51 51 89 89 89 T T F <--- 28 560-579 42 42 42 76 76 76 T T F <--- 29 580-599 48 48 48 84 84 84 T T F <--- Max gen: 52, Max flush: 90 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 88 88 88 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 52 52 52 90 90 90 F F F 4 080-099 52 52 52 90 90 90 F F F 5 100-119 44 44 44 79 79 79 F F F 6 120-139 49 49 49 87 87 87 F F F 7 140-159 48 48 48 85 85 85 F F F 8 160-179 52 52 52 90 90 90 F F F 9 180-199 52 52 52 90 90 90 F F F 10 200-219 47 47 47 82 82 82 F F F 11 220-239 52 52 52 90 90 90 F F F 12 240-259 53 53 53 91 91 91 F F F 13 260-279 53 53 53 91 91 91 F F F 14 280-299 53 53 53 91 91 91 F F F 15 300-319 52 52 52 90 90 90 F F F 16 320-339 48 48 48 85 85 85 F F F 17 340-359 51 51 51 89 89 89 F F F 18 360-379 51 51 51 89 89 89 F F F 19 380-399 48 48 48 85 85 85 F F F 20 400-419 52 52 52 90 90 90 F F F 21 420-439 52 52 52 90 90 90 F F F 22 440-459 48 48 48 86 86 86 F F F 23 460-479 51 51 51 89 89 89 F F F 24 480-499 45 45 45 80 80 80 F F F 25 500-519 46 46 46 81 81 81 F F F 26 520-539 46 46 46 81 81 81 F F F 27 540-559 53 53 53 91 91 91 F F F 28 560-579 53 53 53 91 91 91 F F F 29 580-599 53 53 53 91 91 91 F F F Max gen: 53, Max flush: 91 Begin loop 27 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.451774909Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.49662771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.497459438Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.497516982Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.497536711Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.497563898Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416} {"msg":"Crucible 1df64c5d-48f3-466c-8ad9-35d0dd71ed87 has session id: 4a738459-529b-444f-9d15-a7d55d9ab9f9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498069804Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Upstairs opts: Upstairs UUID: 1df64c5d-48f3-466c-8ad9-35d0dd71ed87, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498128339Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Crucible stats registered with UUID: 1df64c5d-48f3-466c-8ad9-35d0dd71ed87","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498161026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:50.498190304Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498382347Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498431933Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.49846629Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498626266Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"task":"crutest"} {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498659273Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416} {"msg":"1df64c5d-48f3-466c-8ad9-35d0dd71ed87 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498719368Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498741496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"client":"0","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498759914Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"client":"1","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498777293Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"client":"2","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498793971Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.49881135Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498831848Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"ds_connection connected from Ok(127.0.0.1:57533)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498892553Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"ds_connection connected from Ok(127.0.0.1:64468)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498942818Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"ds_connection connected from Ok(127.0.0.1:43950)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.498977965Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.500246735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"client":"0","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.50029478Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"client":"2","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.500868461Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.500941564Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.501437631Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"client":"1","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"[0]R flush_numbers[0..12]: [88, 79, 82, 90, 90, 79, 87, 85, 90, 90, 82, 90]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.501924569Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"[0]R generation[0..12]: [50, 44, 47, 52, 52, 44, 49, 48, 52, 52, 47, 52]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.501954616Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.501975284Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"[1]R flush_numbers[0..12]: [88, 79, 82, 90, 90, 79, 87, 85, 90, 90, 82, 90]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.501994332Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"[1]R generation[0..12]: [50, 44, 47, 52, 52, 44, 49, 48, 52, 52, 47, 52]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502012821Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502032339Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"[2]R flush_numbers[0..12]: [88, 79, 82, 90, 90, 79, 87, 85, 90, 90, 82, 90]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502051078Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"[2]R generation[0..12]: [50, 44, 47, 52, 52, 44, 49, 48, 52, 52, 47, 52]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502069466Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502089174Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:17:50.502105813Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502127221Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Next flush: 92","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.50214351Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502159988Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"":"downstairs","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502176956Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502195615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"client":"0","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502212713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"client":"1","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502229622Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"client":"2","":"downstairs","session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.50225582Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502285907Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"1df64c5d-48f3-466c-8ad9-35d0dd71ed87 is now active with session: 4a738459-529b-444f-9d15-a7d55d9ab9f9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502318084Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502337053Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416,"session_id":"4a738459-529b-444f-9d15-a7d55d9ab9f9"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:17:50.50236475Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416} {"msg":"Activated sub_volume 1df64c5d-48f3-466c-8ad9-35d0dd71ed87","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:50.502391598Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 205 len 20480 data: 5 4 4 3 3 02/30 Read block 534 len 28672 03/30 Write block 113 len 32768 data: 5 5 3 3 5 5 4 4 04/30 Read block 334 len 24576 05/30 Write block 483 len 20480 data: 7 8 5 6 6 06/30 Read block 123 len 16384 07/30 Read block 88 len 20480 08/30 Read block 427 len 8192 09/30 Write block 459 len 36864 data: 5 3 5 5 5 5 5 5 6 10/30 Read block 81 len 4096 11/30 Read block 23 len 32768 12/30 Write block 390 len 16384 data: 4 3 2 2 13/30 Write block 271 len 24576 data: 5 8 8 9 7 7 14/30 Read block 591 len 28672 15/30 Read block 380 len 36864 16/30 Write block 441 len 8192 data: 2 3 17/30 Write block 560 len 4096 data: 3 18/30 Write block 418 len 12288 data: 4 5 5 19/30 Read block 171 len 40960 20/30 Write block 407 len 40960 data: 5 6 5 5 6 4 5 6 5 5 21/30 Read block 432 len 8192 22/30 Flush 23/30 Write block 165 len 32768 data: 5 4 2 2 2 4 5 5 24/30 Read block 177 len 36864 25/30 Read block 307 len 28672 26/30 Write block 61 len 4096 data: 4 27/30 Write block 598 len 4096 data: 2 28/30 Write block 434 len 12288 data: 5 4 4 29/30 Read block 200 len 28672 30/30 Read block 220 len 32768 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Read 4 Done Done Done false 1006 Acked Read 5 Done Done Done false 1007 Acked Read 2 Done Done Done false 1008 Acked Write 9 Done Done Done false 1009 Acked Read 1 Done Done Done false 1010 Acked Read 8 Done Done Done false 1011 Acked Write 4 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Write 2 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Write 3 Done Sent Done false 1018 Acked Read 10 Done Sent Done false 1019 Acked Write 10 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Read 9 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 19 1 20 Done 30 11 29 70 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:50.521615093Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1416} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 88 88 88 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 52 52 52 90 90 90 T F T <--- 4 080-099 52 52 52 90 90 90 F F F 5 100-119 54 44 54 92 79 92 F T F <--- 6 120-139 54 49 54 92 87 92 F T F <--- 7 140-159 48 48 48 85 85 85 F F F 8 160-179 52 52 52 90 90 90 T F T <--- 9 180-199 52 52 52 90 90 90 F F F 10 200-219 54 47 54 92 82 92 F T F <--- 11 220-239 52 52 52 90 90 90 F F F 12 240-259 53 53 53 91 91 91 F F F 13 260-279 54 53 54 92 91 92 F F F <--- 14 280-299 53 53 53 91 91 91 F F F 15 300-319 52 52 52 90 90 90 F F F 16 320-339 48 48 48 85 85 85 F F F 17 340-359 51 51 51 89 89 89 F F F 18 360-379 51 51 51 89 89 89 F F F 19 380-399 54 48 54 92 85 92 F F F <--- 20 400-419 54 52 54 92 90 92 F F F <--- 21 420-439 54 52 54 92 90 92 T F T <--- 22 440-459 54 48 54 92 86 92 F T F <--- 23 460-479 54 51 54 92 89 92 F T F <--- 24 480-499 54 45 54 92 80 92 F T F <--- 25 500-519 46 46 46 81 81 81 F F F 26 520-539 46 46 46 81 81 81 F F F 27 540-559 53 53 53 91 91 91 F F F 28 560-579 54 53 54 92 91 92 F F F <--- 29 580-599 53 53 53 91 91 91 T F T <--- Max gen: 54, Max flush: 92 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 88 88 88 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 55 55 55 93 93 93 F F F 4 080-099 52 52 52 90 90 90 F F F 5 100-119 54 54 54 92 92 92 F F F 6 120-139 54 54 54 92 92 92 F F F 7 140-159 48 48 48 85 85 85 F F F 8 160-179 55 55 55 93 93 93 F F F 9 180-199 52 52 52 90 90 90 F F F 10 200-219 54 54 54 92 92 92 F F F 11 220-239 52 52 52 90 90 90 F F F 12 240-259 53 53 53 91 91 91 F F F 13 260-279 54 54 54 92 92 92 F F F 14 280-299 53 53 53 91 91 91 F F F 15 300-319 52 52 52 90 90 90 F F F 16 320-339 48 48 48 85 85 85 F F F 17 340-359 51 51 51 89 89 89 F F F 18 360-379 51 51 51 89 89 89 F F F 19 380-399 54 54 54 92 92 92 F F F 20 400-419 54 54 54 92 92 92 F F F 21 420-439 55 55 55 93 93 93 F F F 22 440-459 54 54 54 92 92 92 F F F 23 460-479 54 54 54 92 92 92 F F F 24 480-499 54 54 54 92 92 92 F F F 25 500-519 46 46 46 81 81 81 F F F 26 520-539 46 46 46 81 81 81 F F F 27 540-559 53 53 53 91 91 91 F F F 28 560-579 54 54 54 92 92 92 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 55, Max flush: 93 Begin loop 28 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.934093784Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.978606925Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.979347341Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.979394597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.979410786Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.979432504Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424} {"msg":"Crucible 08a9bd0b-c810-4915-9ece-91b5a688427c has session id: ef063ae3-6c66-4934-951d-0fd5e014d0d1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.979980666Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Upstairs opts: Upstairs UUID: 08a9bd0b-c810-4915-9ece-91b5a688427c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.98004276Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Crucible stats registered with UUID: 08a9bd0b-c810-4915-9ece-91b5a688427c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980074678Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:54.980104155Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980292569Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980342814Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980377111Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980575224Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"task":"crutest"} {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980614121Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424} {"msg":"08a9bd0b-c810-4915-9ece-91b5a688427c active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980650798Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980683045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"client":"0","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980714532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"client":"1","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980748909Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"client":"2","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980781456Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980817633Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.98085293Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"ds_connection connected from Ok(127.0.0.1:45386)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.980890946Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"ds_connection connected from Ok(127.0.0.1:49774)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.98107849Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"ds_connection connected from Ok(127.0.0.1:41404)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.981119207Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.982277366Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"client":"1","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.982311413Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"client":"2","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.982332001Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"client":"0","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.982904001Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.982933859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"[0]R flush_numbers[0..12]: [88, 79, 82, 93, 90, 92, 92, 85, 93, 90, 92, 90]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.982952187Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"[0]R generation[0..12]: [50, 44, 47, 55, 52, 54, 54, 48, 55, 52, 54, 52]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.982971305Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.982991174Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"[1]R flush_numbers[0..12]: [88, 79, 82, 93, 90, 92, 92, 85, 93, 90, 92, 90]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983024501Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"[1]R generation[0..12]: [50, 44, 47, 55, 52, 54, 54, 48, 55, 52, 54, 52]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983045159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983064897Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"[2]R flush_numbers[0..12]: [88, 79, 82, 93, 90, 92, 92, 85, 93, 90, 92, 90]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983083526Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"[2]R generation[0..12]: [50, 44, 47, 55, 52, 54, 54, 48, 55, 52, 54, 52]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983105553Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983125642Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.98314229Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983163948Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Next flush: 94","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983183847Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:17:54.983200485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"":"downstairs","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983222803Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983239022Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"client":"0","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983256351Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"client":"1","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983273109Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"client":"2","":"downstairs","session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983290117Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983306256Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"08a9bd0b-c810-4915-9ece-91b5a688427c is now active with session: ef063ae3-6c66-4934-951d-0fd5e014d0d1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983324205Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983339793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424,"session_id":"ef063ae3-6c66-4934-951d-0fd5e014d0d1"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.983355742Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424} {"msg":"Activated sub_volume 08a9bd0b-c810-4915-9ece-91b5a688427c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:54.98337206Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 550 len 16384 data: 5 5 5 5 02/30 Read block 319 len 12288 03/30 Write block 504 len 28672 data: 4 5 5 4 4 6 5 04/30 Read block 53 len 32768 05/30 Read block 481 len 16384 06/30 Read block 478 len 12288 07/30 Read block 470 len 24576 08/30 Read block 407 len 36864 09/30 Write block 518 len 28672 data: 2 2 2 3 4 3 4 10/30 Write block 221 len 40960 data: 7 7 5 6 5 4 5 5 4 4 11/30 Flush 12/30 Write block 332 len 4096 data: 5 13/30 Read block 488 len 32768 14/30 Read block 255 len 16384 15/30 Read block 477 len 32768 16/30 Read block 410 len 8192 17/30 Write block 492 len 12288 data: 4 4 4 18/30 Write block 533 len 4096 data: 6 19/30 Write block 350 len 12288 data: 6 3 5 20/30 Read block 20 len 20480 21/30 Read block 36 len 12288 22/30 Write block 282 len 24576 data: 4 3 4 5 5 5 23/30 Write block 77 len 40960 data: 9 7 6 8 7 9 10 8 7 6 24/30 Write block 344 len 32768 data: 3 3 3 5 6 6 7 4 25/30 Write block 167 len 40960 data: 3 3 3 5 6 6 6 6 6 8 26/30 Read block 352 len 4096 27/30 Read block 565 len 40960 28/30 Read block 574 len 28672 29/30 Write block 362 len 12288 data: 6 6 7 30/30 Read block 250 len 24576 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 3 Done Done Sent false 1002 Acked Write 7 Done Done Sent false 1003 Acked Read 8 Done Done Sent false 1004 Acked Read 4 Done Done Sent false 1005 Acked Read 3 Done Done Sent false 1006 Acked Read 6 Done Done Sent false 1007 Acked Read 9 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 1 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Write 1 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Read 10 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 3 Done Done Sent false 1029 Acked Read 6 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 29 29 Done 30 30 1 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:55.002640881Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1424} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 88 88 88 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 55 55 55 93 93 93 T T F <--- 4 080-099 52 52 52 90 90 90 T T F <--- 5 100-119 54 54 54 92 92 92 F F F 6 120-139 54 54 54 92 92 92 F F F 7 140-159 48 48 48 85 85 85 F F F 8 160-179 55 55 55 93 93 93 T T F <--- 9 180-199 52 52 52 90 90 90 F F F 10 200-219 54 54 54 92 92 92 F F F 11 220-239 56 56 52 94 94 90 F F F <--- 12 240-259 53 53 53 91 91 91 F F F 13 260-279 54 54 54 92 92 92 F F F 14 280-299 53 53 53 91 91 91 T T F <--- 15 300-319 52 52 52 90 90 90 F F F 16 320-339 48 48 48 85 85 85 T T F <--- 17 340-359 51 51 51 89 89 89 T T F <--- 18 360-379 51 51 51 89 89 89 T T F <--- 19 380-399 54 54 54 92 92 92 F F F 20 400-419 54 54 54 92 92 92 F F F 21 420-439 55 55 55 93 93 93 F F F 22 440-459 54 54 54 92 92 92 F F F 23 460-479 54 54 54 92 92 92 F F F 24 480-499 54 54 54 92 92 92 T T F <--- 25 500-519 56 56 46 94 94 81 F F F <--- 26 520-539 56 56 46 94 94 81 T T F <--- 27 540-559 56 56 53 94 94 91 F F T <--- 28 560-579 54 54 54 92 92 92 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 56, Max flush: 94 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 88 88 88 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 57 57 57 95 95 95 F F F 4 080-099 57 57 57 95 95 95 F F F 5 100-119 54 54 54 92 92 92 F F F 6 120-139 54 54 54 92 92 92 F F F 7 140-159 48 48 48 85 85 85 F F F 8 160-179 57 57 57 95 95 95 F F F 9 180-199 52 52 52 90 90 90 F F F 10 200-219 54 54 54 92 92 92 F F F 11 220-239 56 56 56 94 94 94 F F F 12 240-259 53 53 53 91 91 91 F F F 13 260-279 54 54 54 92 92 92 F F F 14 280-299 57 57 57 95 95 95 F F F 15 300-319 52 52 52 90 90 90 F F F 16 320-339 57 57 57 95 95 95 F F F 17 340-359 57 57 57 95 95 95 F F F 18 360-379 57 57 57 95 95 95 F F F 19 380-399 54 54 54 92 92 92 F F F 20 400-419 54 54 54 92 92 92 F F F 21 420-439 55 55 55 93 93 93 F F F 22 440-459 54 54 54 92 92 92 F F F 23 460-479 54 54 54 92 92 92 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 56 56 56 94 94 94 F F F 26 520-539 57 57 57 95 95 95 F F F 27 540-559 56 56 56 94 94 94 F F F 28 560-579 54 54 54 92 92 92 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 57, Max flush: 95 Begin loop 29 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.418273379Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:59.46278894Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.463315264Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.495103534Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.495934291Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.495991706Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.496011995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.496039532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432} {"msg":"Crucible 5ba35c68-2af0-4a85-a17f-3c9b88966d3d has session id: a5bca826-fc8e-4071-8525-c48dfb1c4c62","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.496656698Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Upstairs opts: Upstairs UUID: 5ba35c68-2af0-4a85-a17f-3c9b88966d3d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.496717023Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Crucible stats registered with UUID: 5ba35c68-2af0-4a85-a17f-3c9b88966d3d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.496746Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:59.496763619Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.496950573Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.49698761Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497028866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497187582Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"task":"crutest"} {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.49720896Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432} {"msg":"5ba35c68-2af0-4a85-a17f-3c9b88966d3d active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497294113Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497314361Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"client":"0","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497331929Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"client":"1","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497348848Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"client":"2","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497365797Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497383055Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497401243Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"ds_connection connected from Ok(127.0.0.1:58527)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497530132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"ds_connection connected from Ok(127.0.0.1:38217)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497581308Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"ds_connection connected from Ok(127.0.0.1:58882)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.497639973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499084467Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"client":"1","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499129093Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"client":"2","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499169729Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"client":"0","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499674885Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499754708Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"[0]R flush_numbers[0..12]: [88, 79, 82, 95, 95, 92, 92, 85, 95, 90, 92, 94]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499862289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"[0]R generation[0..12]: [50, 44, 47, 57, 57, 54, 54, 48, 57, 52, 54, 56]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499887797Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499908625Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"[1]R flush_numbers[0..12]: [88, 79, 82, 95, 95, 92, 92, 85, 95, 90, 92, 94]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499927703Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"[1]R generation[0..12]: [50, 44, 47, 57, 57, 54, 54, 48, 57, 52, 54, 56]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499946192Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.49996577Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"[2]R flush_numbers[0..12]: [88, 79, 82, 95, 95, 92, 92, 85, 95, 90, 92, 94]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.499984418Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"[2]R generation[0..12]: [50, 44, 47, 57, 57, 54, 54, 48, 57, 52, 54, 56]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500002887Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500023025Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500039193Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500055972Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Next flush: 96","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500072461Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500090039Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"":"downstairs","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-12-24T00:17:59.500111737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500133945Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"client":"0","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500151264Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"client":"1","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500167992Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"client":"2","":"downstairs","session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500185171Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500201159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"5ba35c68-2af0-4a85-a17f-3c9b88966d3d is now active with session: a5bca826-fc8e-4071-8525-c48dfb1c4c62","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500218868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500233987Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432,"session_id":"a5bca826-fc8e-4071-8525-c48dfb1c4c62"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500249895Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432} {"msg":"Activated sub_volume 5ba35c68-2af0-4a85-a17f-3c9b88966d3d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:17:59.500263834Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 313 len 8192 data: 3 3 02/30 Read block 100 len 16384 03/30 Read block 490 len 24576 04/30 Read block 184 len 24576 05/30 Read block 449 len 24576 06/30 Read block 171 len 8192 07/30 Write block 457 len 16384 data: 5 5 6 4 08/30 Write block 95 len 20480 data: 6 5 5 6 6 09/30 Read block 386 len 36864 10/30 Flush 11/30 Write block 248 len 20480 data: 6 8 7 7 6 12/30 Read block 23 len 36864 13/30 Flush 14/30 Write block 534 len 4096 data: 5 15/30 Read block 187 len 16384 16/30 Write block 530 len 36864 data: 6 6 6 7 6 4 6 6 6 17/30 Write block 206 len 12288 data: 5 5 4 18/30 Read block 485 len 28672 19/30 Read block 494 len 28672 20/30 Write block 320 len 28672 data: 3 5 5 5 5 5 5 21/30 Read block 538 len 12288 22/30 Flush 23/30 Flush 24/30 Write block 437 len 32768 data: 3 2 2 2 3 4 4 5 25/30 Read block 304 len 12288 26/30 Read block 138 len 4096 27/30 Write block 123 len 24576 data: 7 6 6 5 7 7 28/30 Read block 85 len 8192 29/30 Write block 291 len 24576 data: 6 6 4 4 4 4 30/30 Write block 348 len 8192 data: 7 7 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Read 6 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 2 Sent Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 2 2 32 Done 2 28 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:17:59.520181028Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1432} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 88 88 88 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 57 57 57 95 95 95 F F F 4 080-099 57 58 58 95 96 96 F F F <--- 5 100-119 54 54 54 92 92 92 F F F 6 120-139 54 54 54 92 92 92 F T T <--- 7 140-159 48 48 48 85 85 85 F F F 8 160-179 57 57 57 95 95 95 F F F 9 180-199 52 52 52 90 90 90 F F F 10 200-219 54 58 58 92 98 98 F F F <--- 11 220-239 56 56 56 94 94 94 F F F 12 240-259 53 58 58 91 97 97 F F F <--- 13 260-279 54 54 54 92 92 92 F F F 14 280-299 57 57 57 95 95 95 F T T <--- 15 300-319 52 58 58 90 96 96 T F F <--- 16 320-339 57 58 58 95 98 98 F F F <--- 17 340-359 57 57 57 95 95 95 F T T <--- 18 360-379 57 57 57 95 95 95 F F F 19 380-399 54 54 54 92 92 92 F F F 20 400-419 54 54 54 92 92 92 F F F 21 420-439 55 55 55 93 93 93 F T T <--- 22 440-459 54 58 58 92 96 96 F T T <--- 23 460-479 54 58 58 92 96 96 F F F <--- 24 480-499 57 57 57 95 95 95 F F F 25 500-519 56 56 56 94 94 94 F F F 26 520-539 57 58 58 95 98 98 F F F <--- 27 540-559 56 56 56 94 94 94 F F F 28 560-579 54 54 54 92 92 92 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 58, Max flush: 98 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 88 88 88 F F F 1 020-039 44 44 44 79 79 79 F F F 2 040-059 47 47 47 82 82 82 F F F 3 060-079 57 57 57 95 95 95 F F F 4 080-099 58 58 58 96 96 96 F F F 5 100-119 54 54 54 92 92 92 F F F 6 120-139 59 59 59 99 99 99 F F F 7 140-159 48 48 48 85 85 85 F F F 8 160-179 57 57 57 95 95 95 F F F 9 180-199 52 52 52 90 90 90 F F F 10 200-219 58 58 58 98 98 98 F F F 11 220-239 56 56 56 94 94 94 F F F 12 240-259 58 58 58 97 97 97 F F F 13 260-279 54 54 54 92 92 92 F F F 14 280-299 59 59 59 99 99 99 F F F 15 300-319 58 58 58 96 96 96 F F F 16 320-339 58 58 58 98 98 98 F F F 17 340-359 59 59 59 99 99 99 F F F 18 360-379 57 57 57 95 95 95 F F F 19 380-399 54 54 54 92 92 92 F F F 20 400-419 54 54 54 92 92 92 F F F 21 420-439 59 59 59 99 99 99 F F F 22 440-459 59 59 59 99 99 99 F F F 23 460-479 58 58 58 96 96 96 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 56 56 56 94 94 94 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 56 56 56 94 94 94 F F F 28 560-579 54 54 54 92 92 92 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 59, Max flush: 99 Begin loop 30 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:03.957232928Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.001850309Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.002599534Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.00264735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.002665398Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.002687647Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440} {"msg":"Crucible 3a952d87-dc55-4bfd-92f7-f3b43d000099 has session id: bc42f326-d6e1-463a-b22f-e176b45f0c99","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003290964Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Upstairs opts: Upstairs UUID: 3a952d87-dc55-4bfd-92f7-f3b43d000099, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003350349Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Crucible stats registered with UUID: 3a952d87-dc55-4bfd-92f7-f3b43d000099","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003384236Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:04.003410674Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.00357299Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003613166Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003633464Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003767682Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"task":"crutest"} {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003787681Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440} {"msg":"3a952d87-dc55-4bfd-92f7-f3b43d000099 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003863394Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003885292Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"client":"0","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003904761Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003923389Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"client":"2","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003940857Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.003984754Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.004052628Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"ds_connection connected from Ok(127.0.0.1:43299)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.004097374Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"ds_connection connected from Ok(127.0.0.1:48099)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.004133001Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:04.004288837Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:04.004336153Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:04.00437257Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.004515137Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.004556364Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.005434787Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"client":"2","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.005528949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"client":"0","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.005956422Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:04.006114888Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.006426198Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"ds_connection connected from Ok(127.0.0.1:37559)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.006760468Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.008374058Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"[0]R flush_numbers[0..12]: [88, 79, 82, 95, 96, 92, 99, 85, 95, 90, 98, 94]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.008988854Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"[0]R generation[0..12]: [50, 44, 47, 57, 58, 54, 59, 48, 57, 52, 58, 56]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009022201Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009047939Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"[1]R flush_numbers[0..12]: [88, 79, 82, 95, 96, 92, 99, 85, 95, 90, 98, 94]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009071297Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"[1]R generation[0..12]: [50, 44, 47, 57, 58, 54, 59, 48, 57, 52, 58, 56]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009094275Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009118503Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"[2]R flush_numbers[0..12]: [88, 79, 82, 95, 96, 92, 99, 85, 95, 90, 98, 94]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009141581Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"[2]R generation[0..12]: [50, 44, 47, 57, 58, 54, 59, 48, 57, 52, 58, 56]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009164039Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009187947Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Max found gen is 60","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009208105Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid"Wait for a query_work_queue command to finish before sending IO :1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Generation requested: 60 >= found:60","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009236543Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Next flush: 100","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009256591Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009276809Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"":"downstairs","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009297927Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009320085Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"client":"0","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009341414Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"client":"1","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009361732Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"client":"2","":"downstairs","session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.00937897Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009395209Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 3a952d87-dc55-4bfd-92f7-f3b43d000099 is now active with session: bc42f326-d6e1-463a-b22f-e176b45f0c99","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009415717Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009431266Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440,"session_id":"bc42f326-d6e1-463a-b22f-e176b45f0c99"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009447244Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440} {"msg":"Activated sub_volume 3a952d87-dc55-4bfd-92f7-f3b43d000099","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:14.009461113Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 411 len 28672 data: 7 5 6 7 6 6 3 02/30 Flush 03/30 Write block 507 len 8192 data: 5 5 04/30 Read block 546 len 36864 05/30 Flush 06/30 Write block 403 len 4096 data: 5 07/30 Read block 294 len 4096 08/30 Read block 345 len 16384 09/30 Flush 10/30 Write block 88 len 24576 data: 4 3 3 4 5 5 11/30 Read block 112 len 32768 12/30 Read block 231 len 16384 13/30 Write block 132 len 28672 data: 5 6 5 6 3 3 5 14/30 Write block 19 len 24576 data: 7 5 6 5 7 7 15/30 Flush 16/30 Write block 310 len 32768 data: 4 4 4 4 4 4 4 4 17/30 Write block 97 len 4096 data: 6 18/30 Read block 527 len 4096 19/30 Read block 129 len 28672 20/30 Write block 72 len 32768 data: 3 3 4 6 8 10 8 7 21/30 Read block 14 len 36864 22/30 Read block 588 len 40960 23/30 Read block 1 len 8192 24/30 Write block 384 len 4096 data: 9 25/30 Write block 281 len 12288 data: 5 5 4 26/30 Read block 426 len 12288 27/30 Write block 51 len 4096 data: 3 28/30 Read block 235 len 40960 29/30 Write block 197 len 16384 data: 6 9 8 6 30/30 Write block 441 len 20480 data: 4 5 5 6 7 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 2 Done Sent Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 1 Done Sent Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 6 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 8 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Write 3 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 29 2 33 Done 28 1 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) None Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:14.028696587Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1440} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 50 60 103 88 103 F F F <--- 1 020-039 60 44 60 103 79 103 F F F <--- 2 040-059 47 47 47 82 82 82 T F T <--- 3 060-079 57 57 57 95 95 95 T F T <--- 4 080-099 60 58 60 103 96 103 T F T <--- 5 100-119 54 54 54 92 92 92 F F F 6 120-139 60 59 60 103 99 103 F F F <--- 7 140-159 48 48 48 85 85 85 F F F 8 160-179 57 57 57 95 95 95 F F F 9 180-199 52 52 52 90 90 90 T F T <--- 10 200-219 58 58 58 98 98 98 T F T <--- 11 220-239 56 56 56 94 94 94 F F F 12 240-259 58 58 58 97 97 97 F F F 13 260-279 54 54 54 92 92 92 F F F 14 280-299 59 59 59 99 99 99 T F T <--- 15 300-319 58 58 58 96 96 96 T F T <--- 16 320-339 58 58 58 98 98 98 F F F 17 340-359 59 59 59 99 99 99 F F F 18 360-379 57 57 57 95 95 95 F F F 19 380-399 54 54 54 92 92 92 T F T <--- 20 400-419 60 54 60 102 92 102 F T F <--- 21 420-439 59 59 59 99 99 99 F F F 22 440-459 59 59 59 99 99 99 T F T <--- 23 460-479 58 58 58 96 96 96 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 60 56 60 101 94 101 F F F <--- 26 520-539 58 58 58 98 98 98 F F F 27 540-559 56 56 56 94 94 94 F F F 28 560-579 54 54 54 92 92 92 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 60, Max flush: 103 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 103 103 103 F F F 1 020-039 60 60 60 103 103 103 F F F 2 040-059 61 61 61 104 104 104 F F F 3 060-079 61 61 61 104 104 104 F F F 4 080-099 61 61 61 104 104 104 F F F 5 100-119 54 54 54 92 92 92 F F F 6 120-139 60 60 60 103 103 103 F F F 7 140-159 48 48 48 85 85 85 F F F 8 160-179 57 57 57 95 95 95 F F F 9 180-199 61 61 61 104 104 104 F F F 10 200-219 61 61 61 104 104 104 F F F 11 220-239 56 56 56 94 94 94 F F F 12 240-259 58 58 58 97 97 97 F F F 13 260-279 54 54 54 92 92 92 F F F 14 280-299 61 61 61 104 104 104 F F F 15 300-319 61 61 61 104 104 104 F F F 16 320-339 58 58 58 98 98 98 F F F 17 340-359 59 59 59 99 99 99 F F F 18 360-379 57 57 57 95 95 95 F F F 19 380-399 61 61 61 104 104 104 F F F 20 400-419 60 60 60 102 102 102 F F F 21 420-439 59 59 59 99 99 99 F F F 22 440-459 61 61 61 104 104 104 F F F 23 460-479 58 58 58 96 96 96 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 60 60 60 101 101 101 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 56 56 56 94 94 94 F F F 28 560-579 54 54 54 92 92 92 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 61, Max flush: 104 Begin loop 31 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.450905681Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:18.495624114Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.496033189Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.52688754Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.527537053Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.527588739Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.527608177Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.527635624Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448} {"msg":"Crucible dc01e9c0-004c-42e1-91eb-a22a943c5225 has session id: 46e6ba09-086e-4344-af46-73b8f75b0428","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528168498Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Upstairs opts: Upstairs UUID: dc01e9c0-004c-42e1-91eb-a22a943c5225, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528210954Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Crucible stats registered with UUID: dc01e9c0-004c-42e1-91eb-a22a943c5225","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528231992Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:18.528248401Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528416767Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528454183Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528472842Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528629058Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"task":"crutest"} {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528649156Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448} {"msg":"dc01e9c0-004c-42e1-91eb-a22a943c5225 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.52871612Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528736468Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"client":"0","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528755217Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"client":"1","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528772496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"client":"2","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528796883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.52883021Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528855108Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"ds_connection connected from Ok(127.0.0.1:52901)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528916983Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"ds_connection connected from Ok(127.0.0.1:36044)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.52895387Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"ds_connection connected from Ok(127.0.0.1:33777)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.528984987Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.530202881Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"client":"1","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.530295553Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"client":"2","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.530321621Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"client":"0","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.530865763Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.530963785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"[0]R flush_numbers[0..12]: [103, 103, 104, 104, 104, 92, 103, 85, 95, 104, 104, 94]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.530999091Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"[0]R generation[0..12]: [60, 60, 61, 61, 61, 54, 60, 48, 57, 61, 61, 56]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531019769Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531039728Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"[1]R flush_numbers[0..12]: [103, 103, 104, 104, 104, 92, 103, 85, 95, 104, 104, 94]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531058826Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"[1]R generation[0..12]: [60, 60, 61, 61, 61, 54, 60, 48, 57, 61, 61, 56]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531077395Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531101373Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"[2]R flush_numbers[0..12]: [103, 103, 104, 104, 104, 92, 103, 85, 95, 104, 104, 94]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531121121Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"[2]R generation[0..12]: [60, 60, 61, 61, 61, 54, 60, 48, 57, 61, 61, 56]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531140249Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531160567Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531177066Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531194204Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Next flush: 105","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531210703Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531226942Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"":"downstairs","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-12-24T00:18:18.53124415Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531266408Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"client":"0","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531283377Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"client":"1","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531300165Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"client":"2","":"downstairs","session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531317394Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531333502Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"dc01e9c0-004c-42e1-91eb-a22a943c5225 is now active with session: 46e6ba09-086e-4344-af46-73b8f75b0428","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531351371Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531368439Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448,"session_id":"46e6ba09-086e-4344-af46-73b8f75b0428"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531384688Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448} {"msg":"Activated sub_volume dc01e9c0-004c-42e1-91eb-a22a943c5225","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:18.531398916Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 149 len 32768 data: 3 4 5 4 4 3 4 5 02/30 Read block 124 len 28672 03/30 Flush 04/30 Flush 05/30 Flush 06/30 Write block 395 len 36864 data: 4 6 7 6 6 5 4 4 6 07/30 Write block 567 len 24576 data: 2 3 4 5 4 3 08/30 Write block 204 len 16384 data: 6 6 6 6 09/30 Read block 22 len 28672 10/30 Flush 11/30 Flush 12/30 Write block 168 len 16384 data: 4 4 6 7 13/30 Read block 20 len 24576 14/30 Read block 9 len 40960 15/30 Read block 287 len 4096 16/30 Write block 457 len 36864 data: 6 6 7 5 6 6 6 6 6 17/30 Write block 285 len 4096 data: 6 18/30 Write block 294 len 16384 data: 5 5 5 4 19/30 Write block 211 len 24576 data: 3 5 5 4 5 5 20/30 Read block 567 len 40960 21/30 Read block 67 len 32768 22/30 Write block 326 len 36864 data: 6 5 5 5 4 5 6 5 5 23/30 Write block 148 len 12288 data: 3 4 5 24/30 Read block 489 len 16384 25/30 Write block 283 len 8192 data: 5 5 26/30 Read block 333 len 28672 27/30 Write block 155 len 28672 data: 5 6 6 6 6 7 10 28/30 Read block 203 len 32768 29/30 Write block 542 len 8192 data: 6 5 30/30 Write block 239 len 20480 data: 5 5 6 6 7 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 7 Sent Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Write 6 Sent Done Done false 1007 Acked Write 4 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Read 6 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Write 2 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:18.55017683Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1448} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 103 103 103 F F F 1 020-039 60 60 60 103 103 103 F F F 2 040-059 61 61 61 104 104 104 F F F 3 060-079 61 61 61 104 104 104 F F F 4 080-099 61 61 61 104 104 104 F F F 5 100-119 54 54 54 92 92 92 F F F 6 120-139 60 60 60 103 103 103 F F F 7 140-159 48 62 62 85 105 105 T T T <--- 8 160-179 57 57 57 95 95 95 F T T <--- 9 180-199 61 61 61 104 104 104 F F F 10 200-219 61 62 62 104 108 108 F T T <--- 11 220-239 56 56 56 94 94 94 F T T <--- 12 240-259 58 58 58 97 97 97 F T T <--- 13 260-279 54 54 54 92 92 92 F F F 14 280-299 61 61 61 104 104 104 F T T <--- 15 300-319 61 61 61 104 104 104 F F F 16 320-339 58 58 58 98 98 98 F T T <--- 17 340-359 59 59 59 99 99 99 F F F 18 360-379 57 57 57 95 95 95 F F F 19 380-399 61 62 62 104 108 108 F F F <--- 20 400-419 60 62 62 102 108 108 F F F <--- 21 420-439 59 59 59 99 99 99 F F F 22 440-459 61 61 61 104 104 104 F T T <--- 23 460-479 58 58 58 96 96 96 F T T <--- 24 480-499 57 57 57 95 95 95 F F F 25 500-519 60 60 60 101 101 101 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 56 56 56 94 94 94 F T T <--- 28 560-579 54 62 62 92 108 108 F F F <--- 29 580-599 55 55 55 93 93 93 F F F Max gen: 62, Max flush: 108 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 103 103 103 F F F 1 020-039 60 60 60 103 103 103 F F F 2 040-059 61 61 61 104 104 104 F F F 3 060-079 61 61 61 104 104 104 F F F 4 080-099 61 61 61 104 104 104 F F F 5 100-119 54 54 54 92 92 92 F F F 6 120-139 60 60 60 103 103 103 F F F 7 140-159 63 63 63 109 109 109 F F F 8 160-179 63 63 63 109 109 109 F F F 9 180-199 61 61 61 104 104 104 F F F 10 200-219 63 63 63 109 109 109 F F F 11 220-239 63 63 63 109 109 109 F F F 12 240-259 63 63 63 109 109 109 F F F 13 260-279 54 54 54 92 92 92 F F F 14 280-299 63 63 63 109 109 109 F F F 15 300-319 61 61 61 104 104 104 F F F 16 320-339 63 63 63 109 109 109 F F F 17 340-359 59 59 59 99 99 99 F F F 18 360-379 57 57 57 95 95 95 F F F 19 380-399 62 62 62 108 108 108 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 59 59 59 99 99 99 F F F 22 440-459 63 63 63 109 109 109 F F F 23 460-479 63 63 63 109 109 109 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 60 60 60 101 101 101 F F F 26 520-539 58 58 58 98 98 98 F F F 27 540-559 63 63 63 109 109 109 F F F 28 560-579 62 62 62 108 108 108 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 63, Max flush: 109 Begin loop 32 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.019096613Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.064027818Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.064771373Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.064818989Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.064836528Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.064858455Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456} {"msg":"Crucible f331c204-c1f8-48c6-b55a-3765f29befff has session id: 3d6612db-04d5-4cc8-a931-8348329758c0","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065305247Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Upstairs opts: Upstairs UUID: f331c204-c1f8-48c6-b55a-3765f29befff, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065356342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Crucible stats registered with UUID: f331c204-c1f8-48c6-b55a-3765f29befff","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.06537713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:23.065393509Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065586632Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065624619Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065647167Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065735929Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"task":"crutest"} {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065759737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456} {"msg":"f331c204-c1f8-48c6-b55a-3765f29befff active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065818502Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.06583903Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"client":"0","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065857068Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"client":"1","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065874427Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"client":"2","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065896635Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.065929542Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.06596095Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"ds_connection connected from Ok(127.0.0.1:48079)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.066041532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"ds_connection connected from Ok(127.0.0.1:58072)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.066088108Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"ds_connection connected from Ok(127.0.0.1:64317)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.066126995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.067391115Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"client":"0","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.067523193Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"client":"1","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.068028729Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.06813187Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.068372779Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"client":"2","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"[0]R flush_numbers[0..12]: [103, 103, 104, 104, 104, 92, 103, 109, 109, 104, 109, 109]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.068895974Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"[0]R generation[0..12]: [60, 60, 61, 61, 61, 54, 60, 63, 63, 61, 63, 63]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.068920412Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.06894071Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"[1]R flush_numbers[0..12]: [103, 103, 104, 104, 104, 92, 103, 109, 109, 104, 109, 109]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.068960218Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"[1]R generation[0..12]: [60, 60, 61, 61, 61, 54, 60, 63, 63, 61, 63, 63]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.068978796Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.068998205Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"[2]R flush_numbers[0..12]: [103, 103, 104, 104, 104, 92, 103, 109, 109, 104, 109, 109]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069017273Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"[2]R generation[0..12]: [60, 60, 61, 61, 61, 54, 60, 63, 63, 61, 63, 63]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069035671Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.06905528Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069071518Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069088457Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Next flush: 110","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069104796Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069125294Z","hostnameWait for a query_work_queue command to finish before sending IO ":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"":"downstairs","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069149642Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.06916551Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"client":"0","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069182469Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"client":"1","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069199567Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"client":"2","":"downstairs","session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069217276Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069233804Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"f331c204-c1f8-48c6-b55a-3765f29befff is now active with session: 3d6612db-04d5-4cc8-a931-8348329758c0","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069251663Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069267241Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456,"session_id":"3d6612db-04d5-4cc8-a931-8348329758c0"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.06928321Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456} {"msg":"Activated sub_volume f331c204-c1f8-48c6-b55a-3765f29befff","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:23.069297059Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 198 len 24576 data: 10 9 7 8 6 6 02/30 Read block 419 len 40960 03/30 Flush 04/30 Write block 105 len 20480 data: 6 6 5 6 5 05/30 Read block 82 len 20480 06/30 Read block 508 len 12288 07/30 Write block 230 len 24576 data: 5 5 5 5 4 4 08/30 Write block 511 len 16384 data: 5 5 5 5 09/30 Read block 466 len 20480 10/30 Write block 5 len 36864 data: 6 5 4 4 5 6 4 5 5 11/30 Write block 426 len 16384 data: 6 7 5 3 12/30 Write block 239 len 8192 data: 6 6 13/30 Read block 372 len 36864 14/30 Write block 43 len 4096 data: 6 15/30 Write block 187 len 28672 data: 6 5 6 4 4 3 3 16/30 Read block 237 len 32768 17/30 Write block 519 len 28672 data: 3 3 4 5 4 5 5 18/30 Read block 334 len 32768 19/30 Write block 139 len 36864 data: 5 4 6 6 5 5 6 2 2 20/30 Read block 501 len 12288 21/30 Write block 328 len 16384 data: 6 6 5 6 22/30 Write block 62 len 36864 data: 4 3 5 4 4 3 4 3 3 23/30 Flush 24/30 Flush 25/30 Write block 34 len 28672 data: 2 2 3 3 3 5 6 26/30 Read block 551 len 12288 27/30 Write block 536 len 36864 data: 7 7 7 4 5 6 7 6 4 28/30 Write block 445 len 36864 data: 8 8 9 7 8 10 8 6 6 29/30 Write block 275 len 28672 data: 8 8 7 4 4 5 6 30/30 Read block 503 len 12288 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 5 Done Done Done false 1004 Acked Read 5 Done Done Done false 1005 Acked Read 3 Done Done Done false 1006 Acked Write 6 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Read 5 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 8 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Write 4 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Write 7 Done Done Sent false 1029 Acked Read 3 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 24 25 Done 26 27 3 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) Some(JobId(1002)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:23.090043341Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1456} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 60 111 111 103 F F F <--- 1 020-039 60 60 60 103 103 103 T T F <--- 2 040-059 64 64 61 111 111 104 T T F <--- 3 060-079 64 64 61 111 111 104 F F F <--- 4 080-099 61 61 61 104 104 104 F F F 5 100-119 64 64 54 111 111 92 F F T <--- 6 120-139 64 64 60 111 111 103 F F F <--- 7 140-159 64 64 63 111 111 109 F F F <--- 8 160-179 63 63 63 109 109 109 F F F 9 180-199 64 64 64 111 111 110 F F F <--- 10 200-219 64 64 64 110 110 110 F F F 11 220-239 64 64 63 111 111 109 F F F <--- 12 240-259 64 64 63 111 111 109 F F F <--- 13 260-279 54 54 54 92 92 92 T T F <--- 14 280-299 63 63 63 109 109 109 T T F <--- 15 300-319 61 61 61 104 104 104 F F F 16 320-339 64 64 63 111 111 109 F F F <--- 17 340-359 59 59 59 99 99 99 F F F 18 360-379 57 57 57 95 95 95 F F F 19 380-399 62 62 62 108 108 108 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 64 64 59 111 111 99 F F F <--- 22 440-459 63 63 63 109 109 109 T T F <--- 23 460-479 63 63 63 109 109 109 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 64 64 60 111 111 101 F F F <--- 26 520-539 64 64 58 111 111 98 T T F <--- 27 540-559 63 63 63 109 109 109 T T F <--- 28 560-579 62 62 62 108 108 108 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 64, Max flush: 111 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 111 111 111 F F F 1 020-039 65 65 65 112 112 112 F F F 2 040-059 65 65 65 112 112 112 F F F 3 060-079 64 64 64 111 111 111 F F F 4 080-099 61 61 61 104 104 104 F F F 5 100-119 64 64 64 111 111 111 F F F 6 120-139 64 64 64 111 111 111 F F F 7 140-159 64 64 64 111 111 111 F F F 8 160-179 63 63 63 109 109 109 F F F 9 180-199 64 64 64 111 111 111 F F F 10 200-219 64 64 64 110 110 110 F F F 11 220-239 64 64 64 111 111 111 F F F 12 240-259 64 64 64 111 111 111 F F F 13 260-279 65 65 65 112 112 112 F F F 14 280-299 65 65 65 112 112 112 F F F 15 300-319 61 61 61 104 104 104 F F F 16 320-339 64 64 64 111 111 111 F F F 17 340-359 59 59 59 99 99 99 F F F 18 360-379 57 57 57 95 95 95 F F F 19 380-399 62 62 62 108 108 108 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 64 64 64 111 111 111 F F F 22 440-459 65 65 65 112 112 112 F F F 23 460-479 63 63 63 109 109 109 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 64 64 64 111 111 111 F F F 26 520-539 65 65 65 112 112 112 F F F 27 540-559 65 65 65 112 112 112 F F F 28 560-579 62 62 62 108 108 108 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 65, Max flush: 112 Begin loop 33 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.523813729Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.568546371Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.569307934Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.56935674Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.569378328Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.569403056Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464} {"msg":"Crucible 09792466-47b7-4bcf-81a9-381c330f4357 has session id: c0eda55e-7e46-4096-b6a8-0f88adf0bba4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.56993575Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Upstairs opts: Upstairs UUID: 09792466-47b7-4bcf-81a9-381c330f4357, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570001544Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Crucible stats registered with UUID: 09792466-47b7-4bcf-81a9-381c330f4357","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570036921Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:27.570066558Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570219885Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570265911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570300458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570473963Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"task":"crutest"} {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.5705041Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464} {"msg":"09792466-47b7-4bcf-81a9-381c330f4357 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570563025Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570602162Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"client":"0","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570635709Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"client":"1","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570716362Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"client":"2","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570748929Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570780456Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570807044Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"ds_connection connected from Ok(127.0.0.1:54296)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570833051Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"ds_connection connected from Ok(127.0.0.1:56777)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570861439Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"ds_connection connected from Ok(127.0.0.1:44117)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.570891426Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.572142247Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"client":"0","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.572179384Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"client":"1","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.572822188Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.572862665Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.573823701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"client":"2","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"[0]R flush_numbers[0..12]: [111, 112, 112, 111, 104, 111, 111, 111, 109, 111, 110, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574343376Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"[0]R generation[0..12]: [64, 65, 65, 64, 61, 64, 64, 64, 63, 64, 64, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574379142Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.57440492Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"[1]R flush_numbers[0..12]: [111, 112, 112, 111, 104, 111, 111, 111, 109, 111, 110, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574429908Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"[1]R generation[0..12]: [64, 65, 65, 64, 61, 64, 64, 64, 63, 64, 64, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574474274Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574500562Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"[2]R flush_numbers[0..12]: [111, 112, 112, 111, 104, 111, 111, 111, 109, 111, 110, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.57452444Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"[2]R generation[0..12]: [64, 65, 65, 64, 61, 64, 64, 64, 63, 64, 64, 64]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:18:27.574547538Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574575225Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Max found gen is 66","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574595723Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574616862Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Next flush: 113","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.57463684Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574656808Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"":"downstairs","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574677896Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574696935Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"client":"0","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574717953Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"client":"1","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574739001Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"client":"2","":"downstairs","session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574760219Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574783617Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"09792466-47b7-4bcf-81a9-381c330f4357 is now active with session: c0eda55e-7e46-4096-b6a8-0f88adf0bba4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574809275Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574828883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464,"session_id":"c0eda55e-7e46-4096-b6a8-0f88adf0bba4"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.574848712Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464} {"msg":"Activated sub_volume 09792466-47b7-4bcf-81a9-381c330f4357","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:27.57486647Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 153 len 4096 data: 5 02/30 Write block 354 len 36864 data: 6 7 6 6 6 7 7 7 7 03/30 Write block 388 len 16384 data: 7 5 5 4 04/30 Write block 526 len 8192 data: 4 4 05/30 Write block 539 len 16384 data: 5 6 7 8 06/30 Flush 07/30 Write block 24 len 32768 data: 8 7 8 8 6 6 5 5 08/30 Read block 126 len 4096 09/30 Read block 329 len 36864 10/30 Write block 135 len 12288 data: 7 4 4 11/30 Read block 495 len 4096 12/30 Read block 512 len 28672 13/30 Read block 83 len 36864 14/30 Read block 549 len 4096 15/30 Read block 136 len 8192 16/30 Read block 301 len 40960 17/30 Write block 154 len 4096 data: 4 18/30 Write block 169 len 40960 data: 5 7 8 7 7 7 7 9 7 6 19/30 Read block 233 len 8192 20/30 Read block 162 len 32768 21/30 Read block 125 len 28672 22/30 Write block 408 len 36864 data: 7 6 6 8 6 7 8 7 7 23/30 Write block 176 len 32768 data: 10 8 7 6 7 6 6 5 24/30 Write block 193 len 24576 data: 4 3 4 5 7 11 25/30 Flush 26/30 Write block 376 len 40960 data: 5 5 5 5 4 5 5 5 10 7 27/30 Write block 106 len 12288 data: 7 6 7 28/30 Write block 127 len 4096 data: 8 29/30 Read block 514 len 12288 30/30 Write block 206 len 36864 data: 7 7 5 4 3 4 6 6 5 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Write 2 Done Done Done false 1004 Acked Write 4 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 8 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Read 10 Done Done Sent false 1016 Acked Write 1 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Read 8 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Write 6 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Write 1 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 26 28 Done 29 29 4 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:27.593516555Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1464} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 111 111 111 F F F 1 020-039 66 66 65 114 114 112 F F F <--- 2 040-059 65 65 65 112 112 112 F F F 3 060-079 64 64 64 111 111 111 F F F 4 080-099 61 61 61 104 104 104 F F F 5 100-119 64 64 64 111 111 111 T T F <--- 6 120-139 66 66 64 114 114 111 T T F <--- 7 140-159 66 66 64 114 114 111 F F T <--- 8 160-179 66 66 63 114 114 109 F F F <--- 9 180-199 66 66 64 114 114 111 F F F <--- 10 200-219 64 64 64 110 110 110 T T F <--- 11 220-239 64 64 64 111 111 111 F F F 12 240-259 64 64 64 111 111 111 F F F 13 260-279 65 65 65 112 112 112 F F F 14 280-299 65 65 65 112 112 112 F F F 15 300-319 61 61 61 104 104 104 F F F 16 320-339 64 64 64 111 111 111 F F F 17 340-359 66 66 59 113 113 99 F F T <--- 18 360-379 66 66 57 113 113 95 T T T <--- 19 380-399 66 66 62 113 113 108 T T T <--- 20 400-419 66 66 62 114 114 108 F F F <--- 21 420-439 64 64 64 111 111 111 F F F 22 440-459 65 65 65 112 112 112 F F F 23 460-479 63 63 63 109 109 109 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 64 64 64 111 111 111 F F F 26 520-539 66 66 65 113 113 112 F F T <--- 27 540-559 66 66 65 113 113 112 F F F <--- 28 560-579 62 62 62 108 108 108 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 66, Max flush: 114 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 111 111 111 F F F 1 020-039 66 66 66 114 114 114 F F F 2 040-059 65 65 65 112 112 112 F F F 3 060-079 64 64 64 111 111 111 F F F 4 080-099 61 61 61 104 104 104 F F F 5 100-119 67 67 67 115 115 115 F F F 6 120-139 67 67 67 115 115 115 F F F 7 140-159 66 66 66 114 114 114 F F F 8 160-179 66 66 66 114 114 114 F F F 9 180-199 66 66 66 114 114 114 F F F 10 200-219 67 67 67 115 115 115 F F F 11 220-239 64 64 64 111 111 111 F F F 12 240-259 64 64 64 111 111 111 F F F 13 260-279 65 65 65 112 112 112 F F F 14 280-299 65 65 65 112 112 112 F F F 15 300-319 61 61 61 104 104 104 F F F 16 320-339 64 64 64 111 111 111 F F F 17 340-359 66 66 66 113 113 113 F F F 18 360-379 67 67 67 115 115 115 F F F 19 380-399 67 67 67 115 115 115 F F F 20 400-419 66 66 66 114 114 114 F F F 21 420-439 64 64 64 111 111 111 F F F 22 440-459 65 65 65 112 112 112 F F F 23 460-479 63 63 63 109 109 109 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 64 64 64 111 111 111 F F F 26 520-539 66 66 66 113 113 113 F F F 27 540-559 66 66 66 113 113 113 F F F 28 560-579 62 62 62 108 108 108 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 67, Max flush: 115 Begin loop 34 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.008718109Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.053875124Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.054621489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.054669635Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.054686804Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.054708831Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472} {"msg":"Crucible fd4ef952-04de-4bb2-9459-42e3b523f2fe has session id: 8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.055305479Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Upstairs opts: Upstairs UUID: fd4ef952-04de-4bb2-9459-42e3b523f2fe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.055362065Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Crucible stats registered with UUID: fd4ef952-04de-4bb2-9459-42e3b523f2fe","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.055392122Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:32.05541901Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.05565038Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.055691606Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.055711874Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.055881069Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"task":"crutest"} {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.055918036Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472} {"msg":"fd4ef952-04de-4bb2-9459-42e3b523f2fe active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.05598967Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.056012548Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"client":"0","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.056031156Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"client":"1","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.056048715Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"client":"2","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.056065643Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.056083192Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.056140337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"ds_connection connected from Ok(127.0.0.1:39432)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.056302193Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"ds_connection connected from Ok(127.0.0.1:61126)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.056351978Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"ds_connection connected from Ok(127.0.0.1:34134)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.056389045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.057639366Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"client":"2","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.057786493Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"client":"1","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.0578228Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"client":"0","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058305298Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058410799Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"[0]R flush_numbers[0..12]: [111, 114, 112, 111, 104, 115, 115, 114, 114, 114, 115, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058446416Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"[0]R generation[0..12]: [64, 66, 65, 64, 61, 67, 67, 66, 66, 66, 67, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058466354Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058486572Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"[1]R flush_numbers[0..12]: [111, 114, 112, 111, 104, 115, 115, 114, 114, 114, 115, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058505951Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"[1]R generation[0..12]: [64, 66, 65, 64, 61, 67, 67, 66, 66, 66, 67, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058524489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058544077Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"[2]R flush_numbers[0..12]: [111, 114, 112, 111, 104, 115, 115, 114, 114, 114, 115, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058563276Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"[2]R generation[0..12]: [64, 66, 65, 64, 61, 67, 67, 66, 66, 66, 67, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058581674Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058601362Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058620331Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058647478Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Next flush: 116","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058665197Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058681775Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"":"downstairs","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058699034Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO time":"2025-12-24T00:18:32.058714602Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"client":"0","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.05873858Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"client":"1","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058755939Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"client":"2","":"downstairs","session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058773557Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058789966Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"fd4ef952-04de-4bb2-9459-42e3b523f2fe is now active with session: 8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058808164Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058823653Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472,"session_id":"8f427a2d-ad2b-4a62-ab4a-ef72a0449f5a"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.058840011Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472} {"msg":"Activated sub_volume fd4ef952-04de-4bb2-9459-42e3b523f2fe","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:32.05885417Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 96 len 16384 data: 6 7 7 7 02/30 Read block 404 len 28672 03/30 Read block 385 len 8192 04/30 Write block 101 len 12288 data: 6 5 6 05/30 Flush 06/30 Write block 199 len 28672 data: 10 8 9 7 7 7 7 07/30 Write block 131 len 40960 data: 6 6 7 6 8 5 5 6 6 5 08/30 Read block 465 len 24576 09/30 Read block 490 len 40960 10/30 Write block 213 len 8192 data: 7 6 11/30 Flush 12/30 Write block 307 len 32768 data: 6 4 3 5 5 5 5 5 13/30 Read block 217 len 16384 14/30 Read block 402 len 32768 15/30 Write block 196 len 16384 data: 6 8 12 11 16/30 Read block 13 len 32768 17/30 Write block 436 len 36864 data: 5 4 3 3 3 5 6 6 7 18/30 Write block 86 len 4096 data: 7 19/30 Write block 265 len 24576 data: 3 3 4 4 5 5 20/30 Write block 361 len 36864 data: 8 8 7 8 6 6 5 3 3 21/30 Read block 430 len 16384 22/30 Read block 539 len 4096 23/30 Read block 493 len 28672 24/30 Write block 140 len 20480 data: 6 7 7 6 6 25/30 Read block 227 len 24576 26/30 Write block 128 len 24576 data: 8 7 5 7 7 8 27/30 Write block 166 len 36864 data: 5 4 5 6 8 9 8 8 8 28/30 Read block 27 len 8192 29/30 Write block 170 len 4096 data: 9 30/30 Write block 429 len 12288 data: 4 4 5 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Write 3 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Write 4 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Read 4 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Read 7 Done Sent Done false 1023 Acked Write 5 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Write 1 Done Sent Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) None Some(JobId(1010)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:32.078111952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1472} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 111 111 111 F F F 1 020-039 66 66 66 114 114 114 F F F 2 040-059 65 65 65 112 112 112 F F F 3 060-079 64 64 64 111 111 111 F F F 4 080-099 68 61 68 116 104 116 T T T <--- 5 100-119 68 67 68 116 115 116 F F F <--- 6 120-139 68 67 68 117 115 117 T F T <--- 7 140-159 68 66 68 117 114 117 T F T <--- 8 160-179 66 66 66 114 114 114 T F T <--- 9 180-199 68 66 68 117 114 117 T F T <--- 10 200-219 68 67 68 117 115 117 F F F <--- 11 220-239 64 64 64 111 111 111 F F F 12 240-259 64 64 64 111 111 111 F F F 13 260-279 65 65 65 112 112 112 T F T <--- 14 280-299 65 65 65 112 112 112 F F F 15 300-319 61 61 61 104 104 104 T F T <--- 16 320-339 64 64 64 111 111 111 F F F 17 340-359 66 66 66 113 113 113 F F F 18 360-379 67 67 67 115 115 115 T F T <--- 19 380-399 67 67 67 115 115 115 F F F 20 400-419 66 66 66 114 114 114 F F F 21 420-439 64 64 64 111 111 111 T F T <--- 22 440-459 65 65 65 112 112 112 T F T <--- 23 460-479 63 63 63 109 109 109 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 64 64 64 111 111 111 F F F 26 520-539 66 66 66 113 113 113 F F F 27 540-559 66 66 66 113 113 113 F F F 28 560-579 62 62 62 108 108 108 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 68, Max flush: 117 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 111 111 111 F F F 1 020-039 66 66 66 114 114 114 F F F 2 040-059 65 65 65 112 112 112 F F F 3 060-079 64 64 64 111 111 111 F F F 4 080-099 69 69 69 118 118 118 F F F 5 100-119 68 68 68 116 116 116 F F F 6 120-139 69 69 69 118 118 118 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 69 69 69 118 118 118 F F F 9 180-199 69 69 69 118 118 118 F F F 10 200-219 68 68 68 117 117 117 F F F 11 220-239 64 64 64 111 111 111 F F F 12 240-259 64 64 64 111 111 111 F F F 13 260-279 69 69 69 118 118 118 F F F 14 280-299 65 65 65 112 112 112 F F F 15 300-319 69 69 69 118 118 118 F F F 16 320-339 64 64 64 111 111 111 F F F 17 340-359 66 66 66 113 113 113 F F F 18 360-379 69 69 69 118 118 118 F F F 19 380-399 67 67 67 115 115 115 F F F 20 400-419 66 66 66 114 114 114 F F F 21 420-439 69 69 69 118 118 118 F F F 22 440-459 69 69 69 118 118 118 F F F 23 460-479 63 63 63 109 109 109 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 64 64 64 111 111 111 F F F 26 520-539 66 66 66 113 113 113 F F F 27 540-559 66 66 66 113 113 113 F F F 28 560-579 62 62 62 108 108 108 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 69, Max flush: 118 Begin loop 35 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.48777866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.532754561Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.533556191Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.533609136Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.533634994Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.533662661Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480} {"msg":"Crucible bedb5968-a933-4223-b4e0-c177b6efc0dc has session id: aa3c7338-143e-41b7-b540-2134cf10fcda","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.53414147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Upstairs opts: Upstairs UUID: bedb5968-a933-4223-b4e0-c177b6efc0dc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534201494Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Crucible stats registered with UUID: bedb5968-a933-4223-b4e0-c177b6efc0dc","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534226872Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:36.53424735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534441014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534490329Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534520777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534666654Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"task":"crutest"} {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534693522Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480} {"msg":"bedb5968-a933-4223-b4e0-c177b6efc0dc active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534749227Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534769505Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"client":"0","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534788374Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"client":"1","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534809612Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"client":"2","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.5348269Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534844449Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534862047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"ds_connection connected from Ok(127.0.0.1:49886)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.534996145Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"ds_connection connected from Ok(127.0.0.1:37683)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.535041132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"ds_connection connected from Ok(127.0.0.1:34277)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.535075398Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.536256875Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"client":"1","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.536409292Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"client":"0","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.536915158Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.536955045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.5374653Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"client":"2","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"[0]R flush_numbers[0..12]: [111, 114, 112, 111, 118, 116, 118, 118, 118, 118, 117, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.537964307Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"[0]R generation[0..12]: [64, 66, 65, 64, 69, 68, 69, 69, 69, 69, 68, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.537994124Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538029771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"[1]R flush_numbers[0..12]: [111, 114, 112, 111, 118, 116, 118, 118, 118, 118, 117, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538054409Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"[1]R generation[0..12]: [64, 66, 65, 64, 69, 68, 69, 69, 69, 69, 68, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538073547Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538093015Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"[2]R flush_numbers[0..12]: [111, 114, 112, 111, 118, 116, 118, 118, 118, 118, 117, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538111974Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"[2]R generation[0..12]: [64, 66, 65, 64, 69, 68, 69, 69, 69, 69, 68, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538130342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.53815033Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"Wait for a query_work_queue command to finish before sending IO } {"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538173868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538190867Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Next flush: 119","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538207156Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538223304Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"":"downstairs","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538240523Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538256051Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"client":"0","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.53827313Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"client":"1","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538289938Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"client":"2","":"downstairs","session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538308997Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538328855Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"bedb5968-a933-4223-b4e0-c177b6efc0dc is now active with session: aa3c7338-143e-41b7-b540-2134cf10fcda","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538347403Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538363102Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480,"session_id":"aa3c7338-143e-41b7-b540-2134cf10fcda"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538379221Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480} {"msg":"Activated sub_volume bedb5968-a933-4223-b4e0-c177b6efc0dc","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:36.538393219Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 166 len 4096 data: 6 02/30 Read block 99 len 8192 03/30 Write block 470 len 20480 data: 8 8 7 6 7 04/30 Read block 401 len 12288 05/30 Write block 273 len 16384 data: 9 10 9 9 06/30 Write block 350 len 28672 data: 8 5 6 5 7 8 7 07/30 Read block 8 len 4096 08/30 Write block 90 len 36864 data: 4 5 6 6 6 7 7 8 8 09/30 Read block 281 len 36864 10/30 Read block 239 len 12288 11/30 Read block 533 len 20480 12/30 Flush 13/30 Write block 6 len 8192 data: 6 5 14/30 Write block 386 len 28672 data: 6 6 8 6 6 5 3 15/30 Read block 538 len 40960 16/30 Write block 304 len 8192 data: 6 6 17/30 Read block 318 len 20480 18/30 Read block 410 len 32768 19/30 Write block 437 len 28672 data: 5 4 4 4 6 7 7 20/30 Read block 527 len 12288 21/30 Read block 59 len 12288 22/30 Read block 220 len 24576 23/30 Read block 103 len 8192 24/30 Read block 55 len 8192 25/30 Read block 200 len 12288 26/30 Read block 376 len 32768 27/30 Write block 105 len 8192 data: 7 8 28/30 Read block 74 len 16384 29/30 Read block 435 len 4096 30/30 Read block 582 len 32768 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Write 4 Done Done Done false 1005 Acked Write 7 Done Done Done false 1006 Acked Read 1 Done Done Done false 1007 Acked Write 9 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Read 5 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Read 2 Done Done Sent false 1024 Acked Read 3 Done Done Sent false 1025 Acked Read 8 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Read 8 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 23 23 Done 30 30 7 67 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1011)) Some(JobId(1011)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:36.557262215Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1480} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 111 111 111 T T F <--- 1 020-039 66 66 66 114 114 114 F F F 2 040-059 65 65 65 112 112 112 F F F 3 060-079 64 64 64 111 111 111 F F F 4 080-099 70 70 69 119 119 118 F F F <--- 5 100-119 68 68 68 116 116 116 T T F <--- 6 120-139 69 69 69 118 118 118 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 70 70 69 119 119 118 F F T <--- 9 180-199 69 69 69 118 118 118 F F F 10 200-219 68 68 68 117 117 117 F F F 11 220-239 64 64 64 111 111 111 F F F 12 240-259 64 64 64 111 111 111 F F F 13 260-279 70 70 69 119 119 118 F F T <--- 14 280-299 65 65 65 112 112 112 F F F 15 300-319 69 69 69 118 118 118 T T F <--- 16 320-339 64 64 64 111 111 111 F F F 17 340-359 70 70 66 119 119 113 F F T <--- 18 360-379 69 69 69 118 118 118 F F F 19 380-399 67 67 67 115 115 115 T T F <--- 20 400-419 66 66 66 114 114 114 F F F 21 420-439 69 69 69 118 118 118 T T F <--- 22 440-459 69 69 69 118 118 118 T T F <--- 23 460-479 70 70 63 119 119 109 F F T <--- 24 480-499 57 57 57 95 95 95 F F F 25 500-519 64 64 64 111 111 111 F F F 26 520-539 66 66 66 113 113 113 F F F 27 540-559 66 66 66 113 113 113 F F F 28 560-579 62 62 62 108 108 108 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 70, Max flush: 119 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 120 120 120 F F F 1 020-039 66 66 66 114 114 114 F F F 2 040-059 65 65 65 112 112 112 F F F 3 060-079 64 64 64 111 111 111 F F F 4 080-099 70 70 70 119 119 119 F F F 5 100-119 71 71 71 120 120 120 F F F 6 120-139 69 69 69 118 118 118 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 70 70 70 119 119 119 F F F 9 180-199 69 69 69 118 118 118 F F F 10 200-219 68 68 68 117 117 117 F F F 11 220-239 64 64 64 111 111 111 F F F 12 240-259 64 64 64 111 111 111 F F F 13 260-279 70 70 70 119 119 119 F F F 14 280-299 65 65 65 112 112 112 F F F 15 300-319 71 71 71 120 120 120 F F F 16 320-339 64 64 64 111 111 111 F F F 17 340-359 70 70 70 119 119 119 F F F 18 360-379 69 69 69 118 118 118 F F F 19 380-399 71 71 71 120 120 120 F F F 20 400-419 66 66 66 114 114 114 F F F 21 420-439 71 71 71 120 120 120 F F F 22 440-459 71 71 71 120 120 120 F F F 23 460-479 70 70 70 119 119 119 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 64 64 64 111 111 111 F F F 26 520-539 66 66 66 113 113 113 F F F 27 540-559 66 66 66 113 113 113 F F F 28 560-579 62 62 62 108 108 108 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 71, Max flush: 120 Begin loop 36 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:40.9629287Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:41.007253528Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.00768233Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.039004601Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.039668513Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.039715509Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.039731977Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.039755285Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488} {"msg":"Crucible b16b1a00-bd09-4cc2-b9c8-3b449cfa40a7 has session id: 01e83a05-535b-4bfc-992d-1e0d62ee82db","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.04039568Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Upstairs opts: Upstairs UUID: b16b1a00-bd09-4cc2-b9c8-3b449cfa40a7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.040437556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Crucible stats registered with UUID: b16b1a00-bd09-4cc2-b9c8-3b449cfa40a7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.040460754Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:41.040477642Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.040676655Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.040707222Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.040725581Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.040867678Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"task":"crutest"} {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.040888157Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488} {"msg":"b16b1a00-bd09-4cc2-b9c8-3b449cfa40a7 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.04096715Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.040987968Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"client":"0","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.041005376Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"client":"1","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.041022455Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"client":"2","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.041039323Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.041056862Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.04107414Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"ds_connection connected from Ok(127.0.0.1:43778)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.041211418Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"ds_connection connected from Ok(127.0.0.1:64045)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.041257474Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"ds_connection connected from Ok(127.0.0.1:59328)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.041390533Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.042466509Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"client":"2","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.042584969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"client":"1","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.042623265Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"client":"0","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.042987114Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043205144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"[0]R flush_numbers[0..12]: [120, 114, 112, 111, 119, 120, 118, 118, 119, 118, 117, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043286257Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"[0]R generation[0..12]: [71, 66, 65, 64, 70, 71, 69, 69, 70, 69, 68, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043310406Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043330724Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"[1]R flush_numbers[0..12]: [120, 114, 112, 111, 119, 120, 118, 118, 119, 118, 117, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043350002Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"[1]R generation[0..12]: [71, 66, 65, 64, 70, 71, 69, 69, 70, 69, 68, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.04336854Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043388399Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"[2]R flush_numbers[0..12]: [120, 114, 112, 111, 119, 120, 118, 118, 119, 118, 117, 111]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043407667Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"[2]R generation[0..12]: [71, 66, 65, 64, 70, 71, 69, 69, 70, 69, 68, 64]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043429385Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043449723Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043466152Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.04348312Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Next flush: 121","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043499289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":Wait for a query_work_queue command to finish before sending IO "downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043522187Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"":"downstairs","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043539655Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043555364Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"client":"0","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043572272Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"client":"1","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043588991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"client":"2","":"downstairs","session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.04360611Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043622468Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"b16b1a00-bd09-4cc2-b9c8-3b449cfa40a7 is now active with session: 01e83a05-535b-4bfc-992d-1e0d62ee82db","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043640387Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043655825Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488,"session_id":"01e83a05-535b-4bfc-992d-1e0d62ee82db"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043672004Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488} {"msg":"Activated sub_volume b16b1a00-bd09-4cc2-b9c8-3b449cfa40a7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:41.043686233Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 286 len 20480 data: 6 6 5 3 6 03/30 Write block 537 len 16384 data: 8 8 6 7 04/30 Flush 05/30 Write block 192 len 20480 data: 4 5 4 5 7 06/30 Read block 526 len 28672 07/30 Read block 555 len 32768 08/30 Read block 259 len 40960 09/30 Write block 352 len 4096 data: 7 10/30 Read block 64 len 8192 11/30 Read block 323 len 4096 12/30 Write block 507 len 20480 data: 6 6 7 6 6 13/30 Read block 373 len 12288 14/30 Read block 47 len 40960 15/30 Flush 16/30 Write block 21 len 20480 data: 7 6 8 9 8 17/30 Write block 277 len 20480 data: 8 5 5 6 7 18/30 Write block 62 len 24576 data: 5 4 6 5 5 4 19/30 Read block 208 len 8192 20/30 Write block 161 len 16384 data: 11 8 7 6 21/30 Write block 394 len 12288 data: 3 5 7 22/30 Read block 378 len 36864 23/30 Write block 246 len 28672 data: 7 8 7 9 8 8 7 24/30 Write block 221 len 4096 data: 8 25/30 Read block 318 len 16384 26/30 Write block 566 len 40960 data: 3 3 4 5 6 5 4 4 4 2 27/30 Read block 143 len 32768 28/30 Read block 519 len 24576 29/30 Read block 396 len 28672 30/30 Read block 573 len 16384 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 5 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Read 7 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 1 28 Done 2 29 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:41.063124929Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1488} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 120 120 120 F F F 1 020-039 66 66 66 114 114 114 F T T <--- 2 040-059 65 65 65 112 112 112 F F F 3 060-079 64 64 64 111 111 111 F T T <--- 4 080-099 70 70 70 119 119 119 F F F 5 100-119 71 71 71 120 120 120 F F F 6 120-139 69 69 69 118 118 118 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 70 70 70 119 119 119 F T T <--- 9 180-199 69 72 72 118 123 123 F F F <--- 10 200-219 68 68 68 117 117 117 F F F 11 220-239 64 64 64 111 111 111 F T T <--- 12 240-259 64 64 64 111 111 111 F T T <--- 13 260-279 70 70 70 119 119 119 F T T <--- 14 280-299 65 72 72 112 122 122 T T T <--- 15 300-319 71 71 71 120 120 120 F F F 16 320-339 64 64 64 111 111 111 F F F 17 340-359 70 72 72 119 123 123 F F F <--- 18 360-379 69 69 69 118 118 118 F F F 19 380-399 71 71 71 120 120 120 F T T <--- 20 400-419 66 66 66 114 114 114 F F F 21 420-439 71 71 71 120 120 120 F F F 22 440-459 71 71 71 120 120 120 F F F 23 460-479 70 70 70 119 119 119 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 64 72 72 111 123 123 F F F <--- 26 520-539 66 72 72 113 122 122 T F F <--- 27 540-559 66 72 72 113 122 122 T F F <--- 28 560-579 62 62 62 108 108 108 F T T <--- 29 580-599 55 55 55 93 93 93 F F F Max gen: 72, Max flush: 123 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 71 71 71 120 120 120 F F F 1 020-039 73 73 73 124 124 124 F F F 2 040-059 65 65 65 112 112 112 F F F 3 060-079 73 73 73 124 124 124 F F F 4 080-099 70 70 70 119 119 119 F F F 5 100-119 71 71 71 120 120 120 F F F 6 120-139 69 69 69 118 118 118 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 72 72 72 123 123 123 F F F 10 200-219 68 68 68 117 117 117 F F F 11 220-239 73 73 73 124 124 124 F F F 12 240-259 73 73 73 124 124 124 F F F 13 260-279 73 73 73 124 124 124 F F F 14 280-299 73 73 73 124 124 124 F F F 15 300-319 71 71 71 120 120 120 F F F 16 320-339 64 64 64 111 111 111 F F F 17 340-359 72 72 72 123 123 123 F F F 18 360-379 69 69 69 118 118 118 F F F 19 380-399 73 73 73 124 124 124 F F F 20 400-419 66 66 66 114 114 114 F F F 21 420-439 71 71 71 120 120 120 F F F 22 440-459 71 71 71 120 120 120 F F F 23 460-479 70 70 70 119 119 119 F F F 24 480-499 57 57 57 95 95 95 F F F 25 500-519 72 72 72 123 123 123 F F F 26 520-539 72 72 72 122 122 122 F F F 27 540-559 72 72 72 122 122 122 F F F 28 560-579 73 73 73 124 124 124 F F F 29 580-599 55 55 55 93 93 93 F F F Max gen: 73, Max flush: 124 Begin loop 37 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.514793596Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.559596112Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.560364095Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.56042082Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.560456777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.560485154Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496} {"msg":"Crucible 0c5a94e4-0718-4c9d-9d2a-327bc555b7a5 has session id: 00d5cbda-c843-4944-afd5-493082f9a503","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.560976991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Upstairs opts: Upstairs UUID: 0c5a94e4-0718-4c9d-9d2a-327bc555b7a5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561052165Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Crucible stats registered with UUID: 0c5a94e4-0718-4c9d-9d2a-327bc555b7a5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561078293Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:45.561096491Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561372627Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561422443Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.56145632Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561604737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"task":"crutest"} {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561634764Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496} {"msg":"0c5a94e4-0718-4c9d-9d2a-327bc555b7a5 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561687689Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561718267Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"client":"0","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561738145Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"client":"1","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561761493Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"client":"2","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561831247Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561874693Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.56190708Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"ds_connection connected from Ok(127.0.0.1:46871)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561940677Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"ds_connection connected from Ok(127.0.0.1:51139)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.561974204Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"ds_connection connected from Ok(127.0.0.1:45144)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.56226033Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.563232255Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"client":"0","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.563266932Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"client":"2","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.563896177Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.563936493Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.564621544Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"client":"1","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"[0]R flush_numbers[0..12]: [120, 124, 112, 124, 119, 120, 118, 118, 124, 123, 117, 124]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565133869Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"[0]R generation[0..12]: [71, 73, 65, 73, 70, 71, 69, 69, 73, 72, 68, 73]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565162067Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565183145Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"[1]R flush_numbers[0..12]: [120, 124, 112, 124, 119, 120, 118, 118, 124, 123, 117, 124]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565202353Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"[1]R generation[0..12]: [71, 73, 65, 73, 70, 71, 69, 69, 73, 72, 68, 73]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565220701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.56524026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"[2]R flush_numbers[0..12]: [120, 124, 112, 124, 119, 120, 118, 118, 124, 123, 117, 124]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565259328Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"[2]R generation[0..12]: [71, 73, 65, 73, 70, 71, 69, 69, 73, 72, 68, 73]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565277627Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565296975Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565313213Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565330122Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Next flush: 125","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565346461Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565362559Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"":"downstairs","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:18:45.565379458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565401866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"client":"0","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565420744Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"client":"1","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565437783Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"client":"2","":"downstairs","session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565454951Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.56547114Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"0c5a94e4-0718-4c9d-9d2a-327bc555b7a5 is now active with session: 00d5cbda-c843-4944-afd5-493082f9a503","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565489088Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565504487Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496,"session_id":"00d5cbda-c843-4944-afd5-493082f9a503"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565520526Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496} {"msg":"Activated sub_volume 0c5a94e4-0718-4c9d-9d2a-327bc555b7a5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:45.565534774Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 444 len 36864 data: 8 9 9 10 8 9 11 9 7 02/30 Flush 03/30 Write block 94 len 28672 data: 7 8 8 9 9 8 7 04/30 Flush 05/30 Write block 362 len 8192 data: 9 8 06/30 Write block 183 len 20480 data: 6 4 5 6 7 07/30 Write block 592 len 28672 data: 3 3 3 3 2 2 3 08/30 Write block 33 len 36864 data: 5 3 3 4 4 4 6 7 5 09/30 Write block 69 len 28672 data: 4 4 3 4 4 5 7 10/30 Write block 495 len 8192 data: 6 6 11/30 Write block 199 len 16384 data: 12 9 10 8 12/30 Read block 536 len 36864 13/30 Write block 464 len 32768 data: 7 7 6 7 7 7 9 9 14/30 Write block 208 len 12288 data: 6 5 4 15/30 Write block 101 len 28672 data: 7 6 7 5 8 9 7 16/30 Write block 282 len 12288 data: 6 6 6 17/30 Write block 405 len 12288 data: 4 5 6 18/30 Read block 117 len 20480 19/30 Flush 20/30 Write block 129 len 16384 data: 8 6 8 8 21/30 Write block 269 len 20480 data: 6 6 6 9 10 22/30 Read block 460 len 24576 23/30 Read block 95 len 8192 24/30 Write block 6 len 4096 data: 7 25/30 Read block 281 len 16384 26/30 Flush 27/30 Write block 432 len 40960 data: 6 7 6 5 6 6 5 5 5 7 28/30 Read block 419 len 16384 29/30 Read block 498 len 16384 30/30 Write block 131 len 4096 data: 9 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 7 Done Done Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 2 Done Sent Done false 1005 Acked Write 5 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Write 9 Done Sent Done false 1008 Acked Write 7 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Write 3 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 27 1 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1001)) Some(JobId(1025)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:45.583901334Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1496} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 71 74 128 120 128 F F F <--- 1 020-039 74 73 74 127 124 127 F F F <--- 2 040-059 74 65 74 127 112 127 F F F <--- 3 060-079 74 73 74 127 124 127 F F F <--- 4 080-099 74 70 74 126 119 126 F T F <--- 5 100-119 74 71 74 127 120 127 F T F <--- 6 120-139 74 69 74 128 118 128 T F T <--- 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 74 72 74 127 123 127 F F F <--- 10 200-219 74 68 74 127 117 127 F F F <--- 11 220-239 73 73 73 124 124 124 F F F 12 240-259 73 73 73 124 124 124 F F F 13 260-279 74 73 74 128 124 128 F F F <--- 14 280-299 74 73 74 127 124 127 F F F <--- 15 300-319 71 71 71 120 120 120 F F F 16 320-339 64 64 64 111 111 111 F F F 17 340-359 72 72 72 123 123 123 F F F 18 360-379 74 69 74 127 118 127 F F F <--- 19 380-399 73 73 73 124 124 124 F F F 20 400-419 74 66 74 127 114 127 F F F <--- 21 420-439 71 71 71 120 120 120 T F T <--- 22 440-459 74 74 74 125 125 125 T F T <--- 23 460-479 74 70 74 127 119 127 F F F <--- 24 480-499 74 57 74 127 95 127 F F F <--- 25 500-519 72 72 72 123 123 123 F F F 26 520-539 72 72 72 122 122 122 F F F 27 540-559 72 72 72 122 122 122 F F F 28 560-579 73 73 73 124 124 124 F F F 29 580-599 74 55 74 127 93 127 F F F <--- Max gen: 74, Max flush: 128 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 74 128 128 128 F F F 1 020-039 74 74 74 127 127 127 F F F 2 040-059 74 74 74 127 127 127 F F F 3 060-079 74 74 74 127 127 127 F F F 4 080-099 74 74 74 126 126 126 F F F 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 74 74 74 127 127 127 F F F 10 200-219 74 74 74 127 127 127 F F F 11 220-239 73 73 73 124 124 124 F F F 12 240-259 73 73 73 124 124 124 F F F 13 260-279 74 74 74 128 128 128 F F F 14 280-299 74 74 74 127 127 127 F F F 15 300-319 71 71 71 120 120 120 F F F 16 320-339 64 64 64 111 111 111 F F F 17 340-359 72 72 72 123 123 123 F F F 18 360-379 74 74 74 127 127 127 F F F 19 380-399 73 73 73 124 124 124 F F F 20 400-419 74 74 74 127 127 127 F F F 21 420-439 75 75 75 129 129 129 F F F 22 440-459 75 75 75 129 129 129 F F F 23 460-479 74 74 74 127 127 127 F F F 24 480-499 74 74 74 127 127 127 F F F 25 500-519 72 72 72 123 123 123 F F F 26 520-539 72 72 72 122 122 122 F F F 27 540-559 72 72 72 122 122 122 F F F 28 560-579 73 73 73 124 124 124 F F F 29 580-599 74 74 74 127 127 127 F F F Max gen: 75, Max flush: 129 Begin loop 38 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.013392892Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:50.057339883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.057737568Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.089105335Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.089754548Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.089801794Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.089818103Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.089840231Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504} {"msg":"Crucible 49299849-9485-4915-bfb7-a9663a39d1d6 has session id: d119b022-7873-4bb5-abb9-98272151a4a9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.090374764Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Upstairs opts: Upstairs UUID: 49299849-9485-4915-bfb7-a9663a39d1d6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.09041556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Crucible stats registered with UUID: 49299849-9485-4915-bfb7-a9663a39d1d6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.090434499Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:50.090450888Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.090672948Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.090709675Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.090728393Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.09088262Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"task":"crutest"} {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.090902538Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504} {"msg":"49299849-9485-4915-bfb7-a9663a39d1d6 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.090967852Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.090987741Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"client":"0","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.091005429Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"client":"1","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.091024987Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"client":"2","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.091053985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.091085902Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.091125999Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:39220)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.09122609Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:50007)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.091272266Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:38959)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.091308163Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.092583731Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"client":"1","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.092618648Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"client":"0","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.092639637Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"client":"2","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.09317499Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093272342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"[0]R flush_numbers[0..12]: [128, 127, 127, 127, 126, 127, 129, 118, 124, 127, 127, 124]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.09340177Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"[0]R generation[0..12]: [74, 74, 74, 74, 74, 74, 75, 69, 73, 74, 74, 73]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093425758Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093446156Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"[1]R flush_numbers[0..12]: [128, 127, 127, 127, 126, 127, 129, 118, 124, 127, 127, 124]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093465335Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"[1]R generation[0..12]: [74, 74, 74, 74, 74, 74, 75, 69, 73, 74, 74, 73]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093483963Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093507731Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"[2]R flush_numbers[0..12]: [128, 127, 127, 127, 126, 127, 129, 118, 124, 127, 127, 124]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093527209Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"[2]R generation[0..12]: [74, 74, 74, 74, 74, 74, 75, 69, 73, 74, 74, 73]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093545668Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093565466Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093581875Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093598853Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Next flush: 130","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093615022Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.09363141Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"":"downstairs","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093648579Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093664017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"client":"0","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093681006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"client":Wait for a query_work_queue command to finish before sending IO "1","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093706104Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"client":"2","":"downstairs","session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093723462Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093739611Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"49299849-9485-4915-bfb7-a9663a39d1d6 is now active with session: d119b022-7873-4bb5-abb9-98272151a4a9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093757679Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093774718Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504,"session_id":"d119b022-7873-4bb5-abb9-98272151a4a9"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093791246Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504} {"msg":"Activated sub_volume 49299849-9485-4915-bfb7-a9663a39d1d6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:50.093805345Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 549 len 12288 data: 4 6 6 02/30 Flush 03/30 Write block 314 len 28672 data: 6 5 5 5 4 4 4 04/30 Write block 554 len 12288 data: 4 4 4 05/30 Flush 06/30 Write block 63 len 8192 data: 5 7 07/30 Read block 10 len 28672 08/30 Read block 477 len 28672 09/30 Read block 128 len 20480 10/30 Write block 225 len 4096 data: 6 11/30 Read block 238 len 20480 12/30 Write block 535 len 8192 data: 5 8 13/30 Read block 136 len 4096 14/30 Read block 331 len 28672 15/30 Read block 396 len 12288 16/30 Write block 559 len 40960 data: 3 4 2 3 2 3 3 4 4 5 17/30 Read block 175 len 8192 18/30 Write block 560 len 8192 data: 5 3 19/30 Flush 20/30 Write block 310 len 12288 data: 6 6 6 21/30 Write block 298 len 36864 data: 3 5 5 6 7 7 7 7 6 22/30 Read block 36 len 36864 23/30 Write block 340 len 32768 data: 6 4 4 4 4 4 4 6 24/30 Read block 169 len 40960 25/30 Write block 79 len 32768 data: 8 9 8 10 11 9 8 8 26/30 Write block 406 len 36864 data: 6 7 8 7 7 9 7 8 9 27/30 Read block 260 len 16384 28/30 Read block 513 len 8192 29/30 Write block 300 len 4096 data: 6 30/30 Write block 511 len 36864 data: 7 6 6 6 4 4 4 3 4 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 2 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Read 5 Sent Done Done false 1009 Acked Write 1 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Read 1 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 3 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Read 10 Sent Done Done false 1024 Acked Write 8 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Write 1 Sent Done Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 1 1 27 Done 0 24 24 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:50.11257039Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1504} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 74 128 128 128 F F F 1 020-039 74 74 74 127 127 127 F F F 2 040-059 74 74 74 127 127 127 F F F 3 060-079 74 76 76 127 132 132 F T T <--- 4 080-099 74 74 74 126 126 126 F T T <--- 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 74 74 74 127 127 127 F F F 10 200-219 74 74 74 127 127 127 F F F 11 220-239 73 76 76 124 132 132 F F F <--- 12 240-259 73 73 73 124 124 124 F F F 13 260-279 74 74 74 128 128 128 F F F 14 280-299 74 74 74 127 127 127 F T T <--- 15 300-319 76 76 76 131 131 131 F T T <--- 16 320-339 76 76 76 131 131 131 F F F 17 340-359 72 72 72 123 123 123 F T T <--- 18 360-379 74 74 74 127 127 127 F F F 19 380-399 73 73 73 124 124 124 F F F 20 400-419 74 74 74 127 127 127 F T T <--- 21 420-439 75 75 75 129 129 129 F F F 22 440-459 75 75 75 129 129 129 F F F 23 460-479 74 74 74 127 127 127 F F F 24 480-499 74 74 74 127 127 127 F F F 25 500-519 72 72 72 123 123 123 F T T <--- 26 520-539 72 76 76 122 132 132 F F F <--- 27 540-559 76 76 76 131 132 132 F F F <--- 28 560-579 73 76 76 124 132 132 F F F <--- 29 580-599 74 74 74 127 127 127 F F F Max gen: 76, Max flush: 132 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 74 128 128 128 F F F 1 020-039 74 74 74 127 127 127 F F F 2 040-059 74 74 74 127 127 127 F F F 3 060-079 77 77 77 133 133 133 F F F 4 080-099 77 77 77 133 133 133 F F F 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 74 74 74 127 127 127 F F F 10 200-219 74 74 74 127 127 127 F F F 11 220-239 76 76 76 132 132 132 F F F 12 240-259 73 73 73 124 124 124 F F F 13 260-279 74 74 74 128 128 128 F F F 14 280-299 77 77 77 133 133 133 F F F 15 300-319 77 77 77 133 133 133 F F F 16 320-339 76 76 76 131 131 131 F F F 17 340-359 77 77 77 133 133 133 F F F 18 360-379 74 74 74 127 127 127 F F F 19 380-399 73 73 73 124 124 124 F F F 20 400-419 77 77 77 133 133 133 F F F 21 420-439 75 75 75 129 129 129 F F F 22 440-459 75 75 75 129 129 129 F F F 23 460-479 74 74 74 127 127 127 F F F 24 480-499 74 74 74 127 127 127 F F F 25 500-519 77 77 77 133 133 133 F F F 26 520-539 76 76 76 132 132 132 F F F 27 540-559 76 76 76 132 132 132 F F F 28 560-579 76 76 76 132 132 132 F F F 29 580-599 74 74 74 127 127 127 F F F Max gen: 77, Max flush: 133 Begin loop 39 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.573944742Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.619218777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.620013068Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.620071083Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.620091131Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.620118468Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512} {"msg":"Crucible fcab0ada-a7e7-4e75-98af-f95ef6626ca7 has session id: 79903080-b49a-410e-b35f-b4934df859e5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.620665031Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Upstairs opts: Upstairs UUID: fcab0ada-a7e7-4e75-98af-f95ef6626ca7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.620709737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Crucible stats registered with UUID: fcab0ada-a7e7-4e75-98af-f95ef6626ca7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.620728675Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:54.620744794Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.620992872Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621030789Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621049838Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621296666Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"task":"crutest"} {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621320214Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512} {"msg":"fcab0ada-a7e7-4e75-98af-f95ef6626ca7 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621413486Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621448713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"client":"0","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.62148129Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"client":"1","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621516147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"client":"2","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621548884Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621581151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621612788Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:59262)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621728238Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:49628)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.621784603Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37637)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.6218211Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.622882408Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"client":"1","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.622918695Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"client":"0","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623067282Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"client":"2","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623558019Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623598115Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"[0]R flush_numbers[0..12]: [128, 127, 127, 133, 133, 127, 129, 118, 124, 127, 127, 132]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.62366219Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"[0]R generation[0..12]: [74, 74, 74, 77, 77, 74, 75, 69, 73, 74, 74, 76]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623694867Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623716995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"[1]R flush_numbers[0..12]: [128, 127, 127, 133, 133, 127, 129, 118, 124, 127, 127, 132]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623736363Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"[1]R generation[0..12]: [74, 74, 74, 77, 77, 74, 75, 69, 73, 74, 74, 76]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623754962Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.62377481Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"[2]R flush_numbers[0..12]: [128, 127, 127, 133, 133, 127, 129, 118, 124, 127, 127, 132]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623793808Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"[2]R generation[0..12]: [74, 74, 74, 77, 77, 74, 75, 69, 73, 74, 74, 76]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623812037Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623831925Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Max found gen is 78","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623848414Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623871901Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Next flush: 134","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.62388864Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623907838Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"":"downstairs","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623925317Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623940826Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"client":"0","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623958124Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"client":"1","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623975032Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"client":"2","":"downstairs","session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.623992191Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.62400838Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"fcab0ada-a7e7-4e75-98af-f95ef6626ca7 is now active with session: 79903080-b49a-410e-b35f-b4934df859e5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.624026388Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.624041677Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512,"session_id":"79903080-b49a-410e-b35f-b4934df859e5"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.624057995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Activated sub_volume fcab0ada-a7e7-4e75-98af-f95ef6626ca7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:54.624075564Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 211 len 4096 data: 5 02/30 Read block 428 len 36864 03/30 Read block 372 len 4096 04/30 Read block 19 len 12288 05/30 Write block 435 len 16384 data: 6 7 7 6 06/30 Flush 07/30 Write block 46 len 36864 data: 4 3 3 3 4 4 3 4 4 08/30 Flush 09/30 Flush 10/30 Write block 504 len 4096 data: 5 11/30 Write block 582 len 4096 data: 4 12/30 Read block 314 len 32768 13/30 Read block 402 len 24576 14/30 Read block 414 len 20480 15/30 Flush 16/30 Write block 57 len 8192 data: 3 5 17/30 Write block 319 len 24576 data: 5 5 6 6 6 6 18/30 Read block 472 len 28672 19/30 Read block 249 len 28672 20/30 Flush 21/30 Write block 3 len 32768 data: 3 3 7 8 6 5 6 7 22/30 Read block 382 len 36864 23/30 Read block 364 len 16384 24/30 Read block 42 len 24576 25/30 Write block 520 len 24576 data: 4 5 6 5 6 6 26/30 Write block 224 len 36864 data: 7 7 5 6 6 5 6 6 6 27/30 Write block 573 len 40960 data: 5 5 3 2 4 7 6 7 5 5 28/30 Read block 278 len 28672 29/30 Write block 95 len 16384 data: 9 9 10 10 30/30 Read block 130 len 40960 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 3 Done Done Sent false 1004 Acked Write 4 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 9 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Read 8 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 8 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Read 4 Done Done Sent false 1023 Acked Read 6 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 27 28 Done 30 29 3 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:54.643176399Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1512} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 74 74 74 128 128 128 T T F <--- 1 020-039 74 74 74 127 127 127 F F F 2 040-059 78 78 74 138 138 127 F F F <--- 3 060-079 77 77 77 133 133 133 F F F 4 080-099 77 77 77 133 133 133 T T F <--- 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 74 74 74 127 127 127 F F F 10 200-219 78 78 74 134 134 127 F F T <--- 11 220-239 76 76 76 132 132 132 T T F <--- 12 240-259 73 73 73 124 124 124 F F F 13 260-279 74 74 74 128 128 128 F F F 14 280-299 77 77 77 133 133 133 F F F 15 300-319 78 78 77 138 138 133 F F F <--- 16 320-339 78 78 76 138 138 131 F F F <--- 17 340-359 77 77 77 133 133 133 F F F 18 360-379 74 74 74 127 127 127 F F F 19 380-399 73 73 73 124 124 124 F F F 20 400-419 77 77 77 133 133 133 F F F 21 420-439 78 78 75 134 134 129 F F F <--- 22 440-459 75 75 75 129 129 129 F F F 23 460-479 74 74 74 127 127 127 F F F 24 480-499 74 74 74 127 127 127 F F F 25 500-519 78 78 77 137 137 133 F F F <--- 26 520-539 76 76 76 132 132 132 T T F <--- 27 540-559 76 76 76 132 132 132 F F F 28 560-579 76 76 76 132 132 132 T T F <--- 29 580-599 78 78 74 137 137 127 T T F <--- Max gen: 78, Max flush: 138 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 79 79 79 139 139 139 F F F 1 020-039 74 74 74 127 127 127 F F F 2 040-059 78 78 78 138 138 138 F F F 3 060-079 77 77 77 133 133 133 F F F 4 080-099 79 79 79 139 139 139 F F F 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 74 74 74 127 127 127 F F F 10 200-219 78 78 78 134 134 134 F F F 11 220-239 79 79 79 139 139 139 F F F 12 240-259 73 73 73 124 124 124 F F F 13 260-279 74 74 74 128 128 128 F F F 14 280-299 77 77 77 133 133 133 F F F 15 300-319 78 78 78 138 138 138 F F F 16 320-339 78 78 78 138 138 138 F F F 17 340-359 77 77 77 133 133 133 F F F 18 360-379 74 74 74 127 127 127 F F F 19 380-399 73 73 73 124 124 124 F F F 20 400-419 77 77 77 133 133 133 F F F 21 420-439 78 78 78 134 134 134 F F F 22 440-459 75 75 75 129 129 129 F F F 23 460-479 74 74 74 127 127 127 F F F 24 480-499 74 74 74 127 127 127 F F F 25 500-519 78 78 78 137 137 137 F F F 26 520-539 79 79 79 139 139 139 F F F 27 540-559 76 76 76 132 132 132 F F F 28 560-579 79 79 79 139 139 139 F F F 29 580-599 79 79 79 139 139 139 F F F Max gen: 79, Max flush: 139 Begin loop 40 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.053438924Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.09800652Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.098786492Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.098834588Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.098854406Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.098882093Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520} {"msg":"Crucible deb0eb52-0789-464b-9be1-ad1c98cf9361 has session id: 717e8786-dabd-4dd1-b967-2bd6932bd0eb","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.099440845Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Upstairs opts: Upstairs UUID: deb0eb52-0789-464b-9be1-ad1c98cf9361, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.09950041Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Crucible stats registered with UUID: deb0eb52-0789-464b-9be1-ad1c98cf9361","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.099525857Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:59.099546316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.099780525Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.099825901Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.099849479Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100033013Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"task":"crutest"} {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100055511Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520} {"msg":"deb0eb52-0789-464b-9be1-ad1c98cf9361 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100113526Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100135764Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"client":"0","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100153773Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"client":"1","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100170931Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"client":"2","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100197319Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100229346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100257584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:60348)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100358775Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:54374)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100405541Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:56029)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.100440558Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.101711957Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"client":"0","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.101756693Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"client":"2","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.102338642Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.102430004Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.103657367Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"client":"1","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"[0]R flush_numbers[0..12]: [139, 127, 138, 133, 139, 127, 129, 118, 124, 127, 134, 139]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.10420295Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"[0]R generation[0..12]: [79, 74, 78, 77, 79, 74, 75, 69, 73, 74, 78, 79]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104230107Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104251256Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"[1]R flush_numbers[0..12]: [139, 127, 138, 133, 139, 127, 129, 118, 124, 127, 134, 139]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104270584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"[1]R generation[0..12]: [79, 74, 78, 77, 79, 74, 75, 69, 73, 74, 78, 79]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104289362Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.10431257Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"[2]R flush_numbers[0..12]: [139, 127, 138, 133, 139, 127, 129, 118, 124, 127, 134, 139]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104332018Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"[2]R generation[0..12]: [79, 74, 78, 77, 79, 74, 75, 69, 73, 74, 78, 79]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104350757Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104370425Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104387264Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104404892Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Next flush: 140","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104421231Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104437559Z","hostnameWait for a query_work_queue command to finish before sending IO ":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"":"downstairs","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104461047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104476756Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"client":"0","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104493755Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"client":"1","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104510653Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"client":"2","":"downstairs","session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104527761Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.10454415Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"deb0eb52-0789-464b-9be1-ad1c98cf9361 is now active with session: 717e8786-dabd-4dd1-b967-2bd6932bd0eb","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104565928Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104583147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520,"session_id":"717e8786-dabd-4dd1-b967-2bd6932bd0eb"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104599335Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520} {"msg":"Activated sub_volume deb0eb52-0789-464b-9be1-ad1c98cf9361","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:18:59.104613374Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 53 len 8192 data: 5 5 02/30 Read block 265 len 8192 03/30 Read block 534 len 36864 04/30 Write block 255 len 28672 data: 6 7 8 6 7 7 6 05/30 Read block 11 len 36864 06/30 Read block 83 len 8192 07/30 Read block 295 len 40960 08/30 Flush 09/30 Flush 10/30 Write block 77 len 12288 data: 11 9 9 11/30 Flush 12/30 Write block 306 len 40960 data: 7 7 5 4 7 7 7 6 7 6 13/30 Read block 197 len 16384 14/30 Read block 206 len 12288 15/30 Read block 226 len 36864 16/30 Write block 439 len 40960 data: 6 6 8 8 8 9 10 10 11 9 17/30 Read block 382 len 28672 18/30 Write block 194 len 16384 data: 5 6 8 9 19/30 Write block 214 len 20480 data: 7 6 6 5 6 20/30 Read block 157 len 16384 21/30 Read block 243 len 24576 22/30 Write block 227 len 24576 data: 7 7 6 7 7 7 23/30 Write block 571 len 24576 data: 6 5 6 6 4 3 24/30 Write block 42 len 32768 data: 6 7 5 5 5 4 4 4 25/30 Write block 214 len 36864 data: 8 7 7 6 7 6 7 9 8 26/30 Flush 27/30 Write block 420 len 40960 data: 6 5 5 5 5 5 7 8 6 5 28/30 Read block 142 len 20480 29/30 Write block 25 len 36864 data: 9 9 9 7 7 6 6 4 6 30/30 Read block 41 len 32768 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Sent Done false 1001 Acked Read 2 Done Sent Done false 1002 Acked Read 9 Done Sent Done false 1003 Acked Write 7 Done Sent Done false 1004 Acked Read 9 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Read 3 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Read 7 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 9 Done Sent Done false 1029 Acked Read 8 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 0 31 Done 29 0 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:18:59.125748962Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1520} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 79 79 79 139 139 139 F F F 1 020-039 74 74 74 127 127 127 T F T <--- 2 040-059 80 78 80 143 138 143 F F F <--- 3 060-079 80 77 80 142 133 142 F F F <--- 4 080-099 79 79 79 139 139 139 F F F 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 80 74 80 143 127 143 F F F <--- 10 200-219 80 78 80 143 134 143 F F F <--- 11 220-239 80 79 80 143 139 143 F F F <--- 12 240-259 80 73 80 140 124 140 F F F <--- 13 260-279 80 74 80 140 128 140 F F F <--- 14 280-299 77 77 77 133 133 133 F F F 15 300-319 80 78 80 143 138 143 F F F <--- 16 320-339 78 78 78 138 138 138 F F F 17 340-359 77 77 77 133 133 133 F F F 18 360-379 74 74 74 127 127 127 F F F 19 380-399 73 73 73 124 124 124 F F F 20 400-419 77 77 77 133 133 133 F F F 21 420-439 80 78 80 143 134 143 T F T <--- 22 440-459 80 75 80 143 129 143 F F F <--- 23 460-479 74 74 74 127 127 127 F F F 24 480-499 74 74 74 127 127 127 F F F 25 500-519 78 78 78 137 137 137 F F F 26 520-539 79 79 79 139 139 139 F F F 27 540-559 76 76 76 132 132 132 F F F 28 560-579 80 79 80 143 139 143 F F F <--- 29 580-599 79 79 79 139 139 139 F F F Max gen: 80, Max flush: 143 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 79 79 79 139 139 139 F F F 1 020-039 81 81 81 144 144 144 F F F 2 040-059 80 80 80 143 143 143 F F F 3 060-079 80 80 80 142 142 142 F F F 4 080-099 79 79 79 139 139 139 F F F 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 80 80 80 143 143 143 F F F 10 200-219 80 80 80 143 143 143 F F F 11 220-239 80 80 80 143 143 143 F F F 12 240-259 80 80 80 140 140 140 F F F 13 260-279 80 80 80 140 140 140 F F F 14 280-299 77 77 77 133 133 133 F F F 15 300-319 80 80 80 143 143 143 F F F 16 320-339 78 78 78 138 138 138 F F F 17 340-359 77 77 77 133 133 133 F F F 18 360-379 74 74 74 127 127 127 F F F 19 380-399 73 73 73 124 124 124 F F F 20 400-419 77 77 77 133 133 133 F F F 21 420-439 81 81 81 144 144 144 F F F 22 440-459 80 80 80 143 143 143 F F F 23 460-479 74 74 74 127 127 127 F F F 24 480-499 74 74 74 127 127 127 F F F 25 500-519 78 78 78 137 137 137 F F F 26 520-539 79 79 79 139 139 139 F F F 27 540-559 76 76 76 132 132 132 F F F 28 560-579 80 80 80 143 143 143 F F F 29 580-599 79 79 79 139 139 139 F F F Max gen: 81, Max flush: 144 Begin loop 41 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.532922006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:03.576308395Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.57671465Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.607793471Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.608574393Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.60862003Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.608635628Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.608657856Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528} {"msg":"Crucible 51a6e308-ed31-4dca-b566-f8dfb27d16d4 has session id: a347ff83-8c0c-46a1-b935-481b827d4c9c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609146264Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Upstairs opts: Upstairs UUID: 51a6e308-ed31-4dca-b566-f8dfb27d16d4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609201679Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Crucible stats registered with UUID: 51a6e308-ed31-4dca-b566-f8dfb27d16d4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609236636Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:03.609263563Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609454677Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609492653Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609513132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.60964922Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"task":"crutest"} {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609674078Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528} {"msg":"51a6e308-ed31-4dca-b566-f8dfb27d16d4 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609745741Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.60976647Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"client":"0","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609785548Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"client":"1","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609802876Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"client":"2","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609819455Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609836903Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.609909017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"ds_connection connected from Ok(127.0.0.1:63598)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.610054214Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"ds_connection connected from Ok(127.0.0.1:55341)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.610174824Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"ds_connection connected from Ok(127.0.0.1:34755)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.610199012Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.611505098Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"client":"1","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.611543775Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"client":"2","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.611566563Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"client":"0","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612130124Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612159721Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"[0]R flush_numbers[0..12]: [139, 144, 143, 142, 139, 127, 129, 118, 124, 143, 143, 143]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612178339Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"[0]R generation[0..12]: [79, 81, 80, 80, 79, 74, 75, 69, 73, 80, 80, 80]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612197558Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612217356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"[1]R flush_numbers[0..12]: [139, 144, 143, 142, 139, 127, 129, 118, 124, 143, 143, 143]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612237854Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"[1]R generation[0..12]: [79, 81, 80, 80, 79, 74, 75, 69, 73, 80, 80, 80]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612256513Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612276401Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"[2]R flush_numbers[0..12]: [139, 144, 143, 142, 139, 127, 129, 118, 124, 143, 143, 143]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612295769Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"[2]R generation[0..12]: [79, 81, 80, 80, 79, 74, 75, 69, 73, 80, 80, 80]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612314097Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612333726Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612350184Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612370663Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Next flush: 145","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-12-24T00:19:03.612387601Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612410209Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"":"downstairs","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612427558Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612442936Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"client":"0","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612459985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"client":"1","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612477083Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"client":"2","":"downstairs","session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612494162Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.6125104Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"51a6e308-ed31-4dca-b566-f8dfb27d16d4 is now active with session: a347ff83-8c0c-46a1-b935-481b827d4c9c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612528489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612543868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528,"session_id":"a347ff83-8c0c-46a1-b935-481b827d4c9c"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612560046Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528} {"msg":"Activated sub_volume 51a6e308-ed31-4dca-b566-f8dfb27d16d4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:03.612574615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 181 len 12288 data: 7 7 7 02/30 Write block 29 len 28672 data: 8 7 7 5 7 4 4 03/30 Read block 203 len 36864 04/30 Flush 05/30 Write block 305 len 28672 data: 8 8 8 6 5 8 8 06/30 Write block 479 len 36864 data: 3 5 5 7 8 9 6 7 7 07/30 Read block 83 len 32768 08/30 Read block 37 len 8192 09/30 Write block 431 len 40960 data: 6 7 8 7 7 8 8 7 7 7 10/30 Write block 396 len 28672 data: 8 8 7 7 6 5 5 11/30 Read block 584 len 40960 12/30 Read block 199 len 24576 13/30 Read block 400 len 24576 14/30 Flush 15/30 Write block 216 len 4096 data: 8 16/30 Write block 537 len 4096 data: 9 17/30 Read block 164 len 8192 18/30 Read block 391 len 16384 19/30 Write block 465 len 24576 data: 8 7 8 8 8 10 20/30 Read block 291 len 24576 21/30 Write block 540 len 40960 data: 8 8 9 7 5 4 5 4 5 5 22/30 Write block 415 len 32768 data: 8 8 4 5 6 7 6 6 23/30 Read block 397 len 36864 24/30 Write block 427 len 12288 data: 9 7 6 25/30 Read block 60 len 40960 26/30 Write block 337 len 20480 data: 6 5 5 7 5 27/30 Write block 521 len 40960 data: 6 7 6 7 7 5 5 4 4 7 28/30 Write block 260 len 16384 data: 8 7 7 6 29/30 Read block 172 len 28672 30/30 Write block 501 len 20480 data: 6 4 5 6 6 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 7 Done Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Write 7 Sent Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Read 6 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Write 10 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Write 5 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 1 1 27 Done 1 25 25 51 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1013)) Some(JobId(1013)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:03.632732798Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1528} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 79 79 79 139 139 139 F F F 1 020-039 82 82 82 145 145 145 F F F 2 040-059 80 80 80 143 143 143 F F F 3 060-079 80 80 80 142 142 142 F F F 4 080-099 79 79 79 139 139 139 F F F 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 82 82 82 145 145 145 F F F 10 200-219 80 80 80 143 143 143 F T T <--- 11 220-239 80 80 80 143 143 143 F F F 12 240-259 80 80 80 140 140 140 F F F 13 260-279 80 80 80 140 140 140 F T T <--- 14 280-299 77 77 77 133 133 133 F F F 15 300-319 80 82 82 143 146 146 T F F <--- 16 320-339 78 78 78 138 138 138 F T T <--- 17 340-359 77 77 77 133 133 133 F T T <--- 18 360-379 74 74 74 127 127 127 F F F 19 380-399 73 82 82 124 146 146 F F F <--- 20 400-419 77 82 82 133 146 146 F T T <--- 21 420-439 81 82 82 144 146 146 F T T <--- 22 440-459 80 82 82 143 146 146 F F F <--- 23 460-479 74 82 82 127 146 146 F T T <--- 24 480-499 74 82 82 127 146 146 F F F <--- 25 500-519 78 78 78 137 137 137 F T T <--- 26 520-539 79 79 79 139 139 139 F T T <--- 27 540-559 76 76 76 132 132 132 F T T <--- 28 560-579 80 80 80 143 143 143 F F F 29 580-599 79 79 79 139 139 139 F F F Max gen: 82, Max flush: 146 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 79 79 79 139 139 139 F F F 1 020-039 82 82 82 145 145 145 F F F 2 040-059 80 80 80 143 143 143 F F F 3 060-079 80 80 80 142 142 142 F F F 4 080-099 79 79 79 139 139 139 F F F 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 69 69 69 118 118 118 F F F 8 160-179 73 73 73 124 124 124 F F F 9 180-199 82 82 82 145 145 145 F F F 10 200-219 83 83 83 147 147 147 F F F 11 220-239 80 80 80 143 143 143 F F F 12 240-259 80 80 80 140 140 140 F F F 13 260-279 83 83 83 147 147 147 F F F 14 280-299 77 77 77 133 133 133 F F F 15 300-319 82 82 82 146 146 146 F F F 16 320-339 83 83 83 147 147 147 F F F 17 340-359 83 83 83 147 147 147 F F F 18 360-379 74 74 74 127 127 127 F F F 19 380-399 82 82 82 146 146 146 F F F 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 82 82 82 146 146 146 F F F 23 460-479 83 83 83 147 147 147 F F F 24 480-499 82 82 82 146 146 146 F F F 25 500-519 83 83 83 147 147 147 F F F 26 520-539 83 83 83 147 147 147 F F F 27 540-559 83 83 83 147 147 147 F F F 28 560-579 80 80 80 143 143 143 F F F 29 580-599 79 79 79 139 139 139 F F F Max gen: 83, Max flush: 147 Begin loop 42 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.081194394Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.12576518Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.12656472Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.126622335Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.126646793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.126674411Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536} {"msg":"Crucible f878d7ad-e10d-414f-af58-56119b8690f7 has session id: caf43393-3654-409c-a689-416232958244","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127243191Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Upstairs opts: Upstairs UUID: f878d7ad-e10d-414f-af58-56119b8690f7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127298526Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Crucible stats registered with UUID: f878d7ad-e10d-414f-af58-56119b8690f7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127320174Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:08.127336493Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127523806Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127567083Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127587031Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127761166Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"task":"crutest"} {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127780944Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536} {"msg":"f878d7ad-e10d-414f-af58-56119b8690f7 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127850268Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127870226Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"client":"0","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127887765Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"client":"1","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127921042Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"client":"2","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127954099Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.127989006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.128021183Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"ds_connection connected from Ok(127.0.0.1:34437)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.128097007Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"ds_connection connected from Ok(127.0.0.1:36770)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.128130354Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"ds_connection connected from Ok(127.0.0.1:55270)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.128164861Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.129472197Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"client":"0","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.129506454Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"client":"1","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.130165276Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.130205433Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131211725Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"client":"2","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"[0]R flush_numbers[0..12]: [139, 145, 143, 142, 139, 127, 129, 118, 124, 145, 147, 143]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131707312Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"[0]R generation[0..12]: [79, 82, 80, 80, 79, 74, 75, 69, 73, 82, 83, 80]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.13173378Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131754878Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"[1]R flush_numbers[0..12]: [139, 145, 143, 142, 139, 127, 129, 118, 124, 145, 147, 143]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131777716Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"[1]R generation[0..12]: [79, 82, 80, 80, 79, 74, 75, 69, 73, 82, 83, 80]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131796584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131816292Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"[2]R flush_numbers[0..12]: [139, 145, 143, 142, 139, 127, 129, 118, 124, 145, 147, 143]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131835491Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"[2]R generation[0..12]: [79, 82, 80, 80, 79, 74, 75, 69, 73, 82, 83, 80]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131854359Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131874058Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Max found gen is 84","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:19:08.131890346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Generation requested: 84 >= found:84","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131912864Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Next flush: 148","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131929323Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131945841Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"":"downstairs","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.13196324Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131978958Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"client":"0","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.131995977Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"client":"1","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.132012945Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"client":"2","":"downstairs","session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.132030104Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.132048232Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"f878d7ad-e10d-414f-af58-56119b8690f7 is now active with session: caf43393-3654-409c-a689-416232958244","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:19:08.132066501Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.132085669Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536,"session_id":"caf43393-3654-409c-a689-416232958244"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.132102138Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536} {"msg":"Activated sub_volume f878d7ad-e10d-414f-af58-56119b8690f7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:08.132116666Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 157 len 28672 data: 7 7 7 8 12 9 8 03/30 Write block 454 len 4096 data: 6 04/30 Write block 225 len 32768 data: 8 6 8 8 7 8 8 8 05/30 Write block 542 len 16384 data: 10 8 6 5 06/30 Read block 76 len 32768 07/30 Write block 352 len 40960 data: 8 6 8 9 8 7 7 8 8 9 08/30 Read block 289 len 4096 09/30 Read block 389 len 32768 10/30 Read block 501 len 12288 11/30 Read block 430 len 28672 12/30 Write block 200 len 8192 data: 10 11 13/30 Read block 504 len 12288 14/30 Read block 236 len 28672 15/30 Read block 424 len 20480 16/30 Read block 554 len 24576 17/30 Read block 329 len 40960 18/30 Read block 6 len 28672 19/30 Write block 346 len 20480 data: 5 7 8 8 9 20/30 Write block 475 len 8192 data: 7 4 21/30 Write block 291 len 20480 data: 7 7 5 6 6 22/30 Write block 288 len 36864 data: 6 4 7 8 8 6 7 7 6 23/30 Write block 146 len 12288 data: 3 3 4 24/30 Read block 586 len 40960 25/30 Write block 294 len 4096 data: 8 26/30 Read block 183 len 12288 27/30 Read block 558 len 32768 28/30 Read block 504 len 28672 29/30 Write block 140 len 28672 data: 7 8 8 7 7 7 4 30/30 Write block 512 len 8192 data: 7 7 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Done Done Sent false 1001 Acked Write 7 Done Done Sent false 1002 Acked Write 1 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Write 4 Done Done Sent false 1005 Acked Read 8 Done Done Sent false 1006 Acked Write 10 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Read 5 Done Done Sent false 1015 Acked Read 6 Done Done Sent false 1016 Acked Read 10 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 30 34 Done 28 28 0 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:08.150282043Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1536} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 79 79 79 139 139 139 F F F 1 020-039 82 82 82 145 145 145 F F F 2 040-059 80 80 80 143 143 143 F F F 3 060-079 80 80 80 142 142 142 F F F 4 080-099 79 79 79 139 139 139 F F F 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 69 69 69 118 118 118 T T F <--- 8 160-179 73 73 73 124 124 124 T T F <--- 9 180-199 82 82 82 145 145 145 F F F 10 200-219 83 83 83 147 147 147 T T F <--- 11 220-239 80 80 80 143 143 143 T T F <--- 12 240-259 80 80 80 140 140 140 F F F 13 260-279 83 83 83 147 147 147 F F F 14 280-299 77 77 77 133 133 133 T T F <--- 15 300-319 82 82 82 146 146 146 F F F 16 320-339 83 83 83 147 147 147 F F F 17 340-359 83 83 83 147 147 147 T T F <--- 18 360-379 74 74 74 127 127 127 T T F <--- 19 380-399 82 82 82 146 146 146 F F F 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 82 82 82 146 146 146 T T F <--- 23 460-479 83 83 83 147 147 147 T T F <--- 24 480-499 82 82 82 146 146 146 F F F 25 500-519 83 83 83 147 147 147 T T F <--- 26 520-539 83 83 83 147 147 147 F F F 27 540-559 83 83 83 147 147 147 T T F <--- 28 560-579 80 80 80 143 143 143 F F F 29 580-599 79 79 79 139 139 139 F F F Max gen: 83, Max flush: 147 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 79 79 79 139 139 139 F F F 1 020-039 82 82 82 145 145 145 F F F 2 040-059 80 80 80 143 143 143 F F F 3 060-079 80 80 80 142 142 142 F F F 4 080-099 79 79 79 139 139 139 F F F 5 100-119 74 74 74 127 127 127 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 84 84 84 148 148 148 F F F 8 160-179 84 84 84 148 148 148 F F F 9 180-199 82 82 82 145 145 145 F F F 10 200-219 84 84 84 148 148 148 F F F 11 220-239 84 84 84 148 148 148 F F F 12 240-259 80 80 80 140 140 140 F F F 13 260-279 83 83 83 147 147 147 F F F 14 280-299 84 84 84 148 148 148 F F F 15 300-319 82 82 82 146 146 146 F F F 16 320-339 83 83 83 147 147 147 F F F 17 340-359 84 84 84 148 148 148 F F F 18 360-379 84 84 84 148 148 148 F F F 19 380-399 82 82 82 146 146 146 F F F 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 84 84 84 148 148 148 F F F 23 460-479 84 84 84 148 148 148 F F F 24 480-499 82 82 82 146 146 146 F F F 25 500-519 84 84 84 148 148 148 F F F 26 520-539 83 83 83 147 147 147 F F F 27 540-559 84 84 84 148 148 148 F F F 28 560-579 80 80 80 143 143 143 F F F 29 580-599 79 79 79 139 139 139 F F F Max gen: 84, Max flush: 148 Begin loop 43 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.555611758Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:12.598726451Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.599130766Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.62972014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.630361114Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.63040649Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.630423049Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.630445277Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544} {"msg":"Crucible 7a8aa586-b544-4e04-a02a-f3ad5d08c48a has session id: 7d59de8c-613d-40d9-a7ef-da8bd31d2ad2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.630955952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Upstairs opts: Upstairs UUID: 7a8aa586-b544-4e04-a02a-f3ad5d08c48a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631009917Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Crucible stats registered with UUID: 7a8aa586-b544-4e04-a02a-f3ad5d08c48a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631038115Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:12.631058593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631276674Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.63132578Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631360737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631583048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"task":"crutest"} {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631618164Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544} {"msg":"7a8aa586-b544-4e04-a02a-f3ad5d08c48a active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631642772Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631662811Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"client":"0","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631690038Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"client":"1","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631727045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"client":"2","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631766971Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631810778Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631845315Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"ds_connection connected from Ok(127.0.0.1:38113)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.631881872Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"ds_connection connected from Ok(127.0.0.1:62657)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.632034138Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"ds_connection connected from Ok(127.0.0.1:62890)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.632089743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.633206626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"client":"1","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.633326486Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"client":"0","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.633359013Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"client":"2","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.633767337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.633923274Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"[0]R flush_numbers[0..12]: [139, 145, 143, 142, 139, 127, 129, 148, 148, 145, 148, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634028314Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"[0]R generation[0..12]: [79, 82, 80, 80, 79, 74, 75, 84, 84, 82, 84, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634054112Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.63407493Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"[1]R flush_numbers[0..12]: [139, 145, 143, 142, 139, 127, 129, 148, 148, 145, 148, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634094209Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"[1]R generation[0..12]: [79, 82, 80, 80, 79, 74, 75, 84, 84, 82, 84, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634112527Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634132036Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"[2]R flush_numbers[0..12]: [139, 145, 143, 142, 139, 127, 129, 148, 148, 145, 148, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634151014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"[2]R generation[0..12]: [79, 82, 80, 80, 79, 74, 75, 84, 84, 82, 84, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634169382Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.63419253Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Max found gen is 85","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634209429Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 86 >= found:85","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634277123Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Next flush: 149","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634298021Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.63431446Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"":"downstairs","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634331478Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634346927Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"client":"0","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634363595Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"client":"1","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634380184Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"client":"2","":"downstairs","session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634397292Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634413261Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"7a8aa586-b544-4e04-a02a-f3ad5d08c48a is now active with session: 7d59de8c-613d-40d9-a7ef-da8bd31d2ad2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.63443101Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634446088Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544,"session_id":"7d59de8c-613d-40d9-a7ef-da8bd31d2ad2"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634462027Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544} {"msg":"Activated sub_volume 7a8aa586-b544-4e04-a02a-f3ad5d08c48a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:12.634475695Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 286 len 4096 data: 7 03/30 Flush 04/30 Flush 05/30 Write block 65 len 28672 data: 6 6 5 5 5 5 4 06/30 Flush 07/30 Write block 489 len 36864 data: 4 4 5 5 5 5 7 7 5 08/30 Write block 97 len 20480 data: 11 11 9 8 8 09/30 Read block 489 len 16384 10/30 Write block 530 len 12288 data: 8 7 7 11/30 Read block 366 len 16384 12/30 Write block 449 len 40960 data: 10 12 10 8 7 7 6 5 7 7 13/30 Write block 331 len 4096 data: 7 14/30 Write block 109 len 16384 data: 6 5 5 4 15/30 Read block 331 len 28672 16/30 Write block 265 len 40960 data: 4 4 5 5 7 7 7 10 11 11 17/30 Write block 261 len 28672 data: 8 8 7 6 5 5 6 18/30 Read block 554 len 16384 19/30 Read block 90 len 28672 20/30 Read block 211 len 40960 21/30 Read block 502 len 28672 22/30 Read block 108 len 28672 23/30 Read block 57 len 20480 24/30 Write block 459 len 28672 data: 8 6 7 7 7 8 9 25/30 Read block 241 len 24576 26/30 Write block 575 len 8192 data: 5 4 27/30 Write block 43 len 20480 data: 8 6 6 6 5 28/30 Read block 201 len 32768 29/30 Write block 147 len 28672 data: 4 5 5 6 6 5 6 30/30 Read block 379 len 36864 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 9 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 7 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Read 5 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Write 7 Sent Done Done false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 1 28 Done 0 27 26 53 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1005)) Some(JobId(1005)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:12.654901975Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1544} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 79 79 79 139 139 139 F F F 1 020-039 82 82 82 145 145 145 F F F 2 040-059 80 80 80 143 143 143 F T T <--- 3 060-079 80 86 86 142 152 152 F F F <--- 4 080-099 79 79 79 139 139 139 F T T <--- 5 100-119 74 74 74 127 127 127 F T T <--- 6 120-139 75 75 75 129 129 129 F F F 7 140-159 84 84 84 148 148 148 F T T <--- 8 160-179 84 84 84 148 148 148 F F F 9 180-199 82 82 82 145 145 145 F F F 10 200-219 84 84 84 148 148 148 F F F 11 220-239 84 84 84 148 148 148 F F F 12 240-259 80 80 80 140 140 140 F F F 13 260-279 83 83 83 147 147 147 F T T <--- 14 280-299 86 86 86 150 150 150 F F F 15 300-319 82 82 82 146 146 146 F F F 16 320-339 83 83 83 147 147 147 F T T <--- 17 340-359 84 84 84 148 148 148 F F F 18 360-379 84 84 84 148 148 148 F F F 19 380-399 82 82 82 146 146 146 F F F 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 84 84 84 148 148 148 F T T <--- 23 460-479 84 84 84 148 148 148 F T T <--- 24 480-499 82 82 82 146 146 146 F T T <--- 25 500-519 84 84 84 148 148 148 F F F 26 520-539 83 83 83 147 147 147 F T T <--- 27 540-559 84 84 84 148 148 148 F F F 28 560-579 80 80 80 143 143 143 F T T <--- 29 580-599 79 79 79 139 139 139 F F F Max gen: 86, Max flush: 152 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 43 Verifying data now No change after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 79 79 79 139 139 139 F F F 1 020-039 82 82 82 145 145 145 F F F 2 040-059 87 87 87 153 153 153 F F F 3 060-079 86 86 86 152 152 152 F F F 4 080-099 87 87 87 153 153 153 F F F 5 100-119 87 87 87 153 153 153 F F F 6 120-139 75 75 75 129 129 129 F F F 7 140-159 87 87 87 153 153 153 F F F 8 160-179 84 84 84 148 148 148 F F F 9 180-199 82 82 82 145 145 145 F F F 10 200-219 84 84 84 148 148 148 F F F 11 220-239 84 84 84 148 148 148 F F F 12 240-259 80 80 80 140 140 140 F F F 13 260-279 87 87 87 153 153 153 F F F 14 280-299 86 86 86 150 150 150 F F F 15 300-319 82 82 82 146 146 146 F F F 16 320-339 87 87 87 153 153 153 F F F 17 340-359 84 84 84 148 148 148 F F F 18 360-379 84 84 84 148 148 148 F F F 19 380-399 82 82 82 146 146 146 F F F 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 87 87 87 153 153 153 F F F 23 460-479 87 87 87 153 153 153 F F F 24 480-499 87 87 87 153 153 153 F F F 25 500-519 84 84 84 148 148 148 F F F 26 520-539 87 87 87 153 153 153 F F F 27 540-559 84 84 84 148 148 148 F F F 28 560-579 87 87 87 153 153 153 F F F 29 580-599 79 79 79 139 139 139 F F F Max gen: 87, Max flush: 153 Begin loop 44 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.094201319Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.140637183Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.141418935Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.14147601Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.141496298Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.141523796Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552} {"msg":"Crucible 018f4edb-48c8-48c3-878b-10eea3e76889 has session id: 8b4467ee-53aa-4876-be96-025fb844cfc7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.141969407Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Upstairs opts: Upstairs UUID: 018f4edb-48c8-48c3-878b-10eea3e76889, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142026212Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Crucible stats registered with UUID: 018f4edb-48c8-48c3-878b-10eea3e76889","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.14205144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:17.142073058Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142317326Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142368512Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.14239113Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142584343Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"task":"crutest"} {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142612151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552} {"msg":"018f4edb-48c8-48c3-878b-10eea3e76889 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142681375Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142718822Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"client":"0","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142753519Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"client":"1","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142785096Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"client":"2","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142815823Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.14285031Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142886207Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"ds_connection connected from Ok(127.0.0.1:64312)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142921934Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"ds_connection connected from Ok(127.0.0.1:59025)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142954891Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"ds_connection connected from Ok(127.0.0.1:36071)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.142986798Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.144301054Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"client":"1","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.144336681Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"client":"0","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.14502943Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.145059158Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.145535276Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"client":"2","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"[0]R flush_numbers[0..12]: [139, 145, 153, 152, 153, 153, 129, 153, 148, 145, 148, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146027433Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"[0]R generation[0..12]: [79, 82, 87, 86, 87, 87, 75, 87, 84, 82, 84, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.14606033Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146081769Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"[1]R flush_numbers[0..12]: [139, 145, 153, 152, 153, 153, 129, 153, 148, 145, 148, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146101227Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"[1]R generation[0..12]: [79, 82, 87, 86, 87, 87, 75, 87, 84, 82, 84, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146119715Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146139293Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"[2]R flush_numbers[0..12]: [139, 145, 153, 152, 153, 153, 129, 153, 148, 145, 148, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146162591Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"[2]R generation[0..12]: [79, 82, 87, 86, 87, 87, 75, 87, 84, 82, 84, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146188659Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146209327Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Max found gen is 88","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:19:17.146225576Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146248384Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Next flush: 154","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146264813Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146281181Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"":"downstairs","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146298239Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146313688Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"client":"0","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146333127Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"client":"1","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146350495Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"client":"2","":"downstairs","session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:19:17.146367713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146385142Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"018f4edb-48c8-48c3-878b-10eea3e76889 is now active with session: 8b4467ee-53aa-4876-be96-025fb844cfc7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146403201Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146418769Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552,"session_id":"8b4467ee-53aa-4876-be96-025fb844cfc7"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146434788Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552} {"msg":"Activated sub_volume 018f4edb-48c8-48c3-878b-10eea3e76889","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:17.146449046Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 294 len 40960 data: 9 8 7 5 4 6 7 7 8 8 02/30 Write block 447 len 24576 data: 12 10 11 13 11 9 03/30 Write block 160 len 32768 data: 9 13 10 9 7 6 7 5 04/30 Write block 4 len 16384 data: 4 8 9 7 05/30 Write block 137 len 24576 data: 6 7 7 8 9 9 06/30 Write block 590 len 12288 data: 4 4 4 07/30 Read block 283 len 8192 08/30 Read block 431 len 32768 09/30 Flush 10/30 Write block 135 len 32768 data: 9 6 7 8 8 9 10 10 11/30 Write block 101 len 4096 data: 9 12/30 Write block 354 len 8192 data: 9 10 13/30 Read block 345 len 32768 14/30 Read block 59 len 28672 15/30 Read block 229 len 32768 16/30 Write block 196 len 36864 data: 9 10 13 13 11 12 9 8 8 17/30 Write block 68 len 8192 data: 6 6 18/30 Flush 19/30 Write block 392 len 8192 data: 4 3 20/30 Flush 21/30 Write block 579 len 8192 data: 7 8 22/30 Read block 529 len 4096 23/30 Read block 466 len 32768 24/30 Write block 326 len 16384 data: 7 6 7 7 25/30 Flush 26/30 Write block 272 len 28672 data: 11 12 12 10 10 9 6 27/30 Write block 454 len 8192 data: 8 7 28/30 Write block 131 len 16384 data: 10 9 9 7 29/30 Read block 389 len 4096 30/30 Read block 100 len 4096 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Sent false 1001 Acked Write 6 Done Done Sent false 1002 Acked Write 8 Done Done Sent false 1003 Acked Write 4 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Read 2 Done Done Sent false 1007 Acked Read 8 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 8 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:17.163688644Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1552} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 79 154 154 139 F F F <--- 1 020-039 82 82 82 145 145 145 F F F 2 040-059 87 87 87 153 153 153 F F F 3 060-079 88 88 86 155 155 152 F F F <--- 4 080-099 87 87 87 153 153 153 F F F 5 100-119 88 88 87 155 155 153 F F F <--- 6 120-139 88 88 75 155 155 129 T T F <--- 7 140-159 88 88 87 155 155 153 F F F <--- 8 160-179 88 88 84 154 154 148 F F F <--- 9 180-199 88 88 82 155 155 145 F F F <--- 10 200-219 88 88 84 155 155 148 F F F <--- 11 220-239 84 84 84 148 148 148 F F F 12 240-259 80 80 80 140 140 140 F F F 13 260-279 87 87 87 153 153 153 T T F <--- 14 280-299 88 88 86 154 154 150 F F F <--- 15 300-319 88 88 82 154 154 146 F F F <--- 16 320-339 88 88 87 157 157 153 F F F <--- 17 340-359 88 88 84 155 155 148 F F F <--- 18 360-379 84 84 84 148 148 148 F F F 19 380-399 88 88 82 156 156 146 F F F <--- 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 88 88 87 154 154 153 T T F <--- 23 460-479 87 87 87 153 153 153 F F F 24 480-499 87 87 87 153 153 153 F F F 25 500-519 84 84 84 148 148 148 F F F 26 520-539 87 87 87 153 153 153 F F F 27 540-559 84 84 84 148 148 148 F F F 28 560-579 88 88 87 157 157 153 F F F <--- 29 580-599 88 88 79 157 157 139 F F F <--- Max gen: 88, Max flush: 157 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 88 154 154 154 F F F 1 020-039 82 82 82 145 145 145 F F F 2 040-059 87 87 87 153 153 153 F F F 3 060-079 88 88 88 155 155 155 F F F 4 080-099 87 87 87 153 153 153 F F F 5 100-119 88 88 88 155 155 155 F F F 6 120-139 89 89 89 158 158 158 F F F 7 140-159 88 88 88 155 155 155 F F F 8 160-179 88 88 88 154 154 154 F F F 9 180-199 88 88 88 155 155 155 F F F 10 200-219 88 88 88 155 155 155 F F F 11 220-239 84 84 84 148 148 148 F F F 12 240-259 80 80 80 140 140 140 F F F 13 260-279 89 89 89 158 158 158 F F F 14 280-299 88 88 88 154 154 154 F F F 15 300-319 88 88 88 154 154 154 F F F 16 320-339 88 88 88 157 157 157 F F F 17 340-359 88 88 88 155 155 155 F F F 18 360-379 84 84 84 148 148 148 F F F 19 380-399 88 88 88 156 156 156 F F F 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 89 89 89 158 158 158 F F F 23 460-479 87 87 87 153 153 153 F F F 24 480-499 87 87 87 153 153 153 F F F 25 500-519 84 84 84 148 148 148 F F F 26 520-539 87 87 87 153 153 153 F F F 27 540-559 84 84 84 148 148 148 F F F 28 560-579 88 88 88 157 157 157 F F F 29 580-599 88 88 88 157 157 157 F F F Max gen: 89, Max flush: 158 Begin loop 45 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.586361197Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.63189554Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.632665612Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.632712388Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.632728917Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.632750635Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560} {"msg":"Crucible 553b4a00-cb4a-46ad-8151-befba23d9c36 has session id: 563f5ad2-9683-410b-b47d-9177fbe4e469","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633229413Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Upstairs opts: Upstairs UUID: 553b4a00-cb4a-46ad-8151-befba23d9c36, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633292048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Crucible stats registered with UUID: 553b4a00-cb4a-46ad-8151-befba23d9c36","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633323305Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:21.633350603Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633581682Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633621049Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633640517Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633794234Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"task":"crutest"} {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633814132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560} {"msg":"553b4a00-cb4a-46ad-8151-befba23d9c36 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633881276Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633901165Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"client":"0","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633918553Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"client":"1","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.633935502Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"client":"2","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.63395452Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.634005585Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.634042402Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"ds_connection connected from Ok(127.0.0.1:38618)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.63407429Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"ds_connection connected from Ok(127.0.0.1:58732)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.634113626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"ds_connection connected from Ok(127.0.0.1:52347)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.634147203Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.635229199Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"client":"0","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.635354958Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"client":"1","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.635611346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"client":"2","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.635817868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.635899011Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"[0]R flush_numbers[0..12]: [154, 145, 153, 155, 153, 155, 158, 155, 154, 155, 155, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636122021Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"[0]R generation[0..12]: [88, 82, 87, 88, 87, 88, 89, 88, 88, 88, 88, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636155228Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636177416Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"[1]R flush_numbers[0..12]: [154, 145, 153, 155, 153, 155, 158, 155, 154, 155, 155, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636197205Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"[1]R generation[0..12]: [88, 82, 87, 88, 87, 88, 89, 88, 88, 88, 88, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636215873Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636235941Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"[2]R flush_numbers[0..12]: [154, 145, 153, 155, 153, 155, 158, 155, 154, 155, 155, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.63625514Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"[2]R generation[0..12]: [88, 82, 87, 88, 87, 88, 89, 88, 88, 88, 88, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636278167Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636298446Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636314804Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636331723Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Next flush: 159","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636348021Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.63636432Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"":"downstairs","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636381469Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636397067Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"client":"0","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636434564Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"client":"1","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636452032Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"client":"2","":"downstairs","session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636469181Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.63648544Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"553b4a00-cb4a-46ad-8151-befba23d9c36 is now active with session: 563f5ad2-9683-410b-b47d-9177fbe4e469","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636503348Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636518686Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560,"session_id":"563f5ad2-9683-410b-b47d-9177fbe4e469"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636534865Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560} {"msg":"Activated sub_volume 553b4a00-cb4a-46ad-8151-befba23d9c36","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:21.636548874Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 516 len 4096 data: 5 02/30 Read block 531 len 20480 03/30 Write block 16 len 16384 data: 5 4 6 8 04/30 Write block 162 len 20480 data: 11 10 8 7 8 05/30 Read block 1 len 8192 06/30 Read block 213 len 24576 07/30 Write block 526 len 16384 data: 6 6 5 5 08/30 Read block 196 len 16384 09/30 Read block 109 len 24576 10/30 Flush 11/30 Write block 487 len 40960 data: 8 5 5 5 6 6 6 6 8 8 12/30 Write block 590 len 40960 data: 5 5 5 4 4 4 3 3 4 2 13/30 Flush 14/30 Write block 130 len 36864 data: 7 11 10 10 8 10 7 8 9 15/30 Read block 123 len 32768 16/30 Read block 86 len 28672 17/30 Write block 480 len 36864 data: 6 6 8 9 10 7 8 9 6 18/30 Read block 315 len 20480 19/30 Read block 400 len 12288 20/30 Write block 550 len 28672 data: 7 7 6 6 5 5 5 21/30 Write block 245 len 36864 data: 5 8 9 8 10 9 9 8 6 22/30 Read block 230 len 28672 23/30 Write block 587 len 12288 data: 6 5 5 24/30 Write block 276 len 40960 data: 11 10 7 6 7 8 7 7 7 7 25/30 Read block 493 len 8192 26/30 Write block 377 len 20480 data: 6 6 6 5 6 27/30 Write block 254 len 40960 data: 5 7 8 9 7 8 9 9 9 8 28/30 Write block 107 len 20480 data: 8 8 7 6 6 29/30 Read block 594 len 16384 30/30 Read block 33 len 8192 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Sent false 1001 Acked Read 5 Done Done Sent false 1002 Acked Write 4 Done Done Sent false 1003 Acked Write 5 Done Done Sent false 1004 Acked Read 2 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Write 4 Done Done Sent false 1007 Acked Read 4 Done Done Sent false 1008 Acked Read 6 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Read 2 Done Done Sent false 1025 Acked Write 5 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) Some(JobId(1012)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:21.656172644Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1560} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 88 159 159 154 F F F <--- 1 020-039 82 82 82 145 145 145 F F F 2 040-059 87 87 87 153 153 153 F F F 3 060-079 88 88 88 155 155 155 F F F 4 080-099 87 87 87 153 153 153 F F F 5 100-119 88 88 88 155 155 155 T T F <--- 6 120-139 89 89 89 158 158 158 T T F <--- 7 140-159 88 88 88 155 155 155 F F F 8 160-179 90 90 88 159 159 154 F F F <--- 9 180-199 88 88 88 155 155 155 F F F 10 200-219 88 88 88 155 155 155 F F F 11 220-239 84 84 84 148 148 148 F F F 12 240-259 80 80 80 140 140 140 T T F <--- 13 260-279 89 89 89 158 158 158 T T F <--- 14 280-299 88 88 88 154 154 154 T T F <--- 15 300-319 88 88 88 154 154 154 F F F 16 320-339 88 88 88 157 157 157 F F F 17 340-359 88 88 88 155 155 155 F F F 18 360-379 84 84 84 148 148 148 T T F <--- 19 380-399 88 88 88 156 156 156 T T F <--- 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 89 89 89 158 158 158 F F F 23 460-479 87 87 87 153 153 153 F F F 24 480-499 90 90 87 160 160 153 T T F <--- 25 500-519 90 90 84 159 159 148 F F F <--- 26 520-539 90 90 87 159 159 153 F F F <--- 27 540-559 84 84 84 148 148 148 T T F <--- 28 560-579 88 88 88 157 157 157 F F F 29 580-599 90 90 88 160 160 157 T T F <--- Max gen: 90, Max flush: 160 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 F F F 1 020-039 82 82 82 145 145 145 F F F 2 040-059 87 87 87 153 153 153 F F F 3 060-079 88 88 88 155 155 155 F F F 4 080-099 87 87 87 153 153 153 F F F 5 100-119 91 91 91 161 161 161 F F F 6 120-139 91 91 91 161 161 161 F F F 7 140-159 88 88 88 155 155 155 F F F 8 160-179 90 90 90 159 159 159 F F F 9 180-199 88 88 88 155 155 155 F F F 10 200-219 88 88 88 155 155 155 F F F 11 220-239 84 84 84 148 148 148 F F F 12 240-259 91 91 91 161 161 161 F F F 13 260-279 91 91 91 161 161 161 F F F 14 280-299 91 91 91 161 161 161 F F F 15 300-319 88 88 88 154 154 154 F F F 16 320-339 88 88 88 157 157 157 F F F 17 340-359 88 88 88 155 155 155 F F F 18 360-379 91 91 91 161 161 161 F F F 19 380-399 91 91 91 161 161 161 F F F 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 89 89 89 158 158 158 F F F 23 460-479 87 87 87 153 153 153 F F F 24 480-499 91 91 91 161 161 161 F F F 25 500-519 90 90 90 159 159 159 F F F 26 520-539 90 90 90 159 159 159 F F F 27 540-559 91 91 91 161 161 161 F F F 28 560-579 88 88 88 157 157 157 F F F 29 580-599 91 91 91 161 161 161 F F F Max gen: 91, Max flush: 161 Begin loop 46 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.072704493Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.118261773Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.119064663Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.119112469Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.119129037Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.119151065Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568} {"msg":"Crucible 346e1431-5cf2-4d6e-9faa-34530292be0b has session id: 2c4caba5-22a2-4048-ab98-10089f66a52d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.11955356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Upstairs opts: Upstairs UUID: 346e1431-5cf2-4d6e-9faa-34530292be0b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.119616075Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Crucible stats registered with UUID: 346e1431-5cf2-4d6e-9faa-34530292be0b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.119639473Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:26.119656511Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.119920628Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.119964494Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.119985342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120144628Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"task":"crutest"} {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120165237Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568} {"msg":"346e1431-5cf2-4d6e-9faa-34530292be0b active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120237551Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120257259Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"client":"0","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120274597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"client":"1","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120291266Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"client":"2","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120307924Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.12036006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120381968Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"ds_connection connected from Ok(127.0.0.1:54894)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120408565Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"ds_connection connected from Ok(127.0.0.1:40164)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120534405Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"ds_connection connected from Ok(127.0.0.1:54039)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.120658684Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.121613431Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"client":"1","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.121656647Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"client":"0","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.121871628Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"client":"2","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122213918Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122287922Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"[0]R flush_numbers[0..12]: [159, 145, 153, 155, 153, 161, 161, 155, 159, 155, 155, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122380134Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"[0]R generation[0..12]: [90, 82, 87, 88, 87, 91, 91, 88, 90, 88, 88, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122406502Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.12242755Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"[1]R flush_numbers[0..12]: [159, 145, 153, 155, 153, 161, 161, 155, 159, 155, 155, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122447038Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"[1]R generation[0..12]: [90, 82, 87, 88, 87, 91, 91, 88, 90, 88, 88, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122465796Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122485245Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"[2]R flush_numbers[0..12]: [159, 145, 153, 155, 153, 161, 161, 155, 159, 155, 155, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122504273Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"[2]R generation[0..12]: [90, 82, 87, 88, 87, 91, 91, 88, 90, 88, 88, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122522921Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.12254259Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-12-24T00:19:26.122558968Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122581216Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Next flush: 162","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122597605Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122613913Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"":"downstairs","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122635631Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.12265178Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"client":"0","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122669028Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"client":"1","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122686177Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"client":"2","":"downstairs","session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122703596Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122720094Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"346e1431-5cf2-4d6e-9faa-34530292be0b is now active with session: 2c4caba5-22a2-4048-ab98-10089f66a52d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122737893Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122753161Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568,"session_id":"2c4caba5-22a2-4048-ab98-10089f66a52d"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.12276986Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568} {"msg":"Activated sub_volume 346e1431-5cf2-4d6e-9faa-34530292be0b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:26.122787458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 360 len 32768 data: 9 10 10 9 9 7 7 6 02/30 Write block 206 len 24576 data: 8 8 7 6 5 6 03/30 Write block 49 len 12288 data: 5 5 5 04/30 Read block 243 len 12288 05/30 Read block 264 len 4096 06/30 Write block 302 len 36864 data: 9 9 8 9 9 9 7 6 9 07/30 Write block 591 len 32768 data: 6 6 5 5 5 4 4 5 08/30 Write block 341 len 12288 data: 6 5 5 09/30 Write block 87 len 24576 data: 6 5 4 5 6 7 10/30 Flush 11/30 Write block 440 len 20480 data: 8 9 9 9 10 12/30 Flush 13/30 Write block 546 len 12288 data: 6 5 6 14/30 Write block 530 len 28672 data: 9 8 8 8 7 6 9 15/30 Read block 335 len 28672 16/30 Read block 136 len 8192 17/30 Write block 536 len 40960 data: 10 10 9 7 9 9 11 9 7 6 18/30 Read block 588 len 24576 19/30 Read block 591 len 20480 20/30 Read block 117 len 8192 21/30 Read block 89 len 28672 22/30 Read block 354 len 4096 23/30 Write block 344 len 24576 data: 5 5 6 8 9 9 24/30 Write block 196 len 32768 data: 10 11 14 14 12 13 10 9 25/30 Write block 491 len 16384 data: 7 7 7 7 26/30 Read block 101 len 36864 27/30 Read block 455 len 24576 28/30 Read block 474 len 32768 29/30 Write block 262 len 4096 data: 10 30/30 Read block 117 len 4096 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Sent false 1001 Acked Write 6 Done Done Sent false 1002 Acked Write 3 Done Done Sent false 1003 Acked Read 3 Done Done Sent false 1004 Acked Read 1 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Write 8 Done Done Sent false 1007 Acked Write 3 Done Done Sent false 1008 Acked Write 6 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 5 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 3 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Write 1 Done Done Sent false 1029 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1011)) Some(JobId(1011)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:26.141678652Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1568} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 F F F 1 020-039 82 82 82 145 145 145 F F F 2 040-059 92 92 87 162 162 153 F F F <--- 3 060-079 88 88 88 155 155 155 F F F 4 080-099 92 92 87 162 162 153 F F F <--- 5 100-119 91 91 91 161 161 161 F F F 6 120-139 91 91 91 161 161 161 F F F 7 140-159 88 88 88 155 155 155 F F F 8 160-179 90 90 90 159 159 159 F F F 9 180-199 88 88 88 155 155 155 T T F <--- 10 200-219 92 92 88 162 162 155 T T F <--- 11 220-239 84 84 84 148 148 148 F F F 12 240-259 91 91 91 161 161 161 F F F 13 260-279 91 91 91 161 161 161 T T F <--- 14 280-299 91 91 91 161 161 161 F F F 15 300-319 92 92 88 162 162 154 F F F <--- 16 320-339 88 88 88 157 157 157 F F F 17 340-359 92 92 88 162 162 155 T T F <--- 18 360-379 92 92 91 162 162 161 F F F <--- 19 380-399 91 91 91 161 161 161 F F F 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 92 92 89 163 163 158 F F F <--- 23 460-479 87 87 87 153 153 153 F F F 24 480-499 91 91 91 161 161 161 T T F <--- 25 500-519 90 90 90 159 159 159 F F F 26 520-539 90 90 90 159 159 159 T T F <--- 27 540-559 91 91 91 161 161 161 T T F <--- 28 560-579 88 88 88 157 157 157 F F F 29 580-599 92 92 91 162 162 161 F F F <--- Max gen: 92, Max flush: 163 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 F F F 1 020-039 82 82 82 145 145 145 F F F 2 040-059 92 92 92 162 162 162 F F F 3 060-079 88 88 88 155 155 155 F F F 4 080-099 92 92 92 162 162 162 F F F 5 100-119 91 91 91 161 161 161 F F F 6 120-139 91 91 91 161 161 161 F F F 7 140-159 88 88 88 155 155 155 F F F 8 160-179 90 90 90 159 159 159 F F F 9 180-199 93 93 93 164 164 164 F F F 10 200-219 93 93 93 164 164 164 F F F 11 220-239 84 84 84 148 148 148 F F F 12 240-259 91 91 91 161 161 161 F F F 13 260-279 93 93 93 164 164 164 F F F 14 280-299 91 91 91 161 161 161 F F F 15 300-319 92 92 92 162 162 162 F F F 16 320-339 88 88 88 157 157 157 F F F 17 340-359 93 93 93 164 164 164 F F F 18 360-379 92 92 92 162 162 162 F F F 19 380-399 91 91 91 161 161 161 F F F 20 400-419 83 83 83 147 147 147 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 92 92 92 163 163 163 F F F 23 460-479 87 87 87 153 153 153 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 90 90 90 159 159 159 F F F 26 520-539 93 93 93 164 164 164 F F F 27 540-559 93 93 93 164 164 164 F F F 28 560-579 88 88 88 157 157 157 F F F 29 580-599 92 92 92 162 162 162 F F F Max gen: 93, Max flush: 164 Begin loop 47 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.553278911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:30.596601866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.5970104Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.627738573Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.628516345Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.628562811Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.628584739Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.628607227Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576} {"msg":"Crucible 4afb6c26-b963-421a-8079-a5a83e36678c has session id: 399cbd99-7809-4866-881f-4fe135ae3cc1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629123672Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Upstairs opts: Upstairs UUID: 4afb6c26-b963-421a-8079-a5a83e36678c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629169568Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Crucible stats registered with UUID: 4afb6c26-b963-421a-8079-a5a83e36678c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629189996Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:30.629207045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629406857Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629445084Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629464352Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629595591Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"task":"crutest"} {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629619129Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576} {"msg":"4afb6c26-b963-421a-8079-a5a83e36678c active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629688723Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629709731Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"client":"0","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629727389Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"client":"1","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629751137Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"client":"2","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629784714Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629812782Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629847809Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"ds_connection connected from Ok(127.0.0.1:50117)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629882916Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"ds_connection connected from Ok(127.0.0.1:37197)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.629915733Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"ds_connection connected from Ok(127.0.0.1:49834)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.62994984Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.631247597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"client":"2","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.631281214Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"client":"1","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.631301952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"client":"0","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.631829336Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.631945416Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"[0]R flush_numbers[0..12]: [159, 145, 162, 155, 162, 161, 161, 155, 159, 164, 164, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.631989362Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"[0]R generation[0..12]: [90, 82, 92, 88, 92, 91, 91, 88, 90, 93, 93, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.63200979Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632029879Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"[1]R flush_numbers[0..12]: [159, 145, 162, 155, 162, 161, 161, 155, 159, 164, 164, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632049017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"[1]R generation[0..12]: [90, 82, 92, 88, 92, 91, 91, 88, 90, 93, 93, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632071065Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632091223Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"[2]R flush_numbers[0..12]: [159, 145, 162, 155, 162, 161, 161, 155, 159, 164, 164, 148]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632110461Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"[2]R generation[0..12]: [90, 82, 92, 88, 92, 91, 91, 88, 90, 93, 93, 84]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.63212885Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632148288Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Max found gen is 94","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632164987Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Generation requested: 94 >= found:94","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632182305Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Next flush: 165","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632198694Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632215223Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"":"downstairs","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632232501Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.63224788Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"client":"0","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632265018Z","hostname"Wait for a query_work_queue command to finish before sending IO :"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"client":"1","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632289026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"client":"2","":"downstairs","session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632306295Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632322483Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"4afb6c26-b963-421a-8079-a5a83e36678c is now active with session: 399cbd99-7809-4866-881f-4fe135ae3cc1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632342311Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.6323579Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576,"session_id":"399cbd99-7809-4866-881f-4fe135ae3cc1"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632373939Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576} {"msg":"Activated sub_volume 4afb6c26-b963-421a-8079-a5a83e36678c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:30.632387807Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 595 len 20480 data: 6 5 5 6 3 03/30 Write block 261 len 32768 data: 10 11 9 7 6 6 7 6 04/30 Write block 30 len 8192 data: 8 8 05/30 Write block 404 len 8192 data: 3 5 06/30 Flush 07/30 Write block 64 len 36864 data: 8 7 7 6 7 7 6 5 5 08/30 Write block 549 len 24576 data: 6 8 8 7 7 6 09/30 Write block 79 len 24576 data: 10 10 9 11 12 10 10/30 Read block 158 len 4096 11/30 Read block 540 len 36864 12/30 Write block 235 len 36864 data: 5 5 4 5 7 7 7 7 8 13/30 Read block 451 len 4096 14/30 Read block 264 len 20480 15/30 Read block 501 len 16384 16/30 Write block 218 len 8192 data: 8 7 17/30 Write block 592 len 28672 data: 7 6 6 7 6 6 7 18/30 Read block 261 len 16384 19/30 Read block 91 len 8192 20/30 Flush 21/30 Write block 369 len 4096 data: 4 22/30 Write block 380 len 20480 data: 6 7 6 6 11 23/30 Read block 406 len 28672 24/30 Write block 62 len 28672 data: 6 6 9 8 8 7 8 25/30 Write block 232 len 12288 data: 9 6 5 26/30 Write block 401 len 40960 data: 6 6 7 4 6 7 8 9 8 8 27/30 Read block 58 len 4096 28/30 Read block 573 len 8192 29/30 Read block 17 len 24576 30/30 Read block 288 len 16384 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 5 Done Done Done false 1002 Acked Write 8 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Write 2 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 9 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Write 6 Sent Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Read 1 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Read 4 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 0 28 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:30.650927222Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1576} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 F F F 1 020-039 82 94 94 145 166 166 F F F <--- 2 040-059 92 92 92 162 162 162 F F F 3 060-079 88 94 94 155 167 167 F T T <--- 4 080-099 92 94 94 162 167 167 F F F <--- 5 100-119 91 91 91 161 161 161 F F F 6 120-139 91 91 91 161 161 161 F F F 7 140-159 88 88 88 155 155 155 F F F 8 160-179 90 90 90 159 159 159 F F F 9 180-199 93 93 93 164 164 164 F F F 10 200-219 93 94 94 164 167 167 F F F <--- 11 220-239 84 94 94 148 167 167 F T T <--- 12 240-259 91 94 94 161 167 167 F F F <--- 13 260-279 93 94 94 164 166 166 F F F <--- 14 280-299 91 91 91 161 161 161 F F F 15 300-319 92 92 92 162 162 162 F F F 16 320-339 88 88 88 157 157 157 F F F 17 340-359 93 93 93 164 164 164 F F F 18 360-379 92 92 92 162 162 162 F T T <--- 19 380-399 91 91 91 161 161 161 F T T <--- 20 400-419 83 94 94 147 166 166 F T T <--- 21 420-439 83 83 83 147 147 147 F F F 22 440-459 92 92 92 163 163 163 F F F 23 460-479 87 87 87 153 153 153 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 90 90 90 159 159 159 F F F 26 520-539 93 93 93 164 164 164 F F F 27 540-559 93 94 94 164 167 167 F F F <--- 28 560-579 88 88 88 157 157 157 F F F 29 580-599 92 94 94 162 167 167 T F F <--- Max gen: 94, Max flush: 167 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 47 Verifying data now No change after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 F F F 1 020-039 94 94 94 166 166 166 F F F 2 040-059 92 92 92 162 162 162 F F F 3 060-079 95 95 95 168 168 168 F F F 4 080-099 94 94 94 167 167 167 F F F 5 100-119 91 91 91 161 161 161 F F F 6 120-139 91 91 91 161 161 161 F F F 7 140-159 88 88 88 155 155 155 F F F 8 160-179 90 90 90 159 159 159 F F F 9 180-199 93 93 93 164 164 164 F F F 10 200-219 94 94 94 167 167 167 F F F 11 220-239 95 95 95 168 168 168 F F F 12 240-259 94 94 94 167 167 167 F F F 13 260-279 94 94 94 166 166 166 F F F 14 280-299 91 91 91 161 161 161 F F F 15 300-319 92 92 92 162 162 162 F F F 16 320-339 88 88 88 157 157 157 F F F 17 340-359 93 93 93 164 164 164 F F F 18 360-379 95 95 95 168 168 168 F F F 19 380-399 95 95 95 168 168 168 F F F 20 400-419 95 95 95 168 168 168 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 92 92 92 163 163 163 F F F 23 460-479 87 87 87 153 153 153 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 90 90 90 159 159 159 F F F 26 520-539 93 93 93 164 164 164 F F F 27 540-559 94 94 94 167 167 167 F F F 28 560-579 88 88 88 157 157 157 F F F 29 580-599 94 94 94 167 167 167 F F F Max gen: 95, Max flush: 168 Begin loop 48 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.090754679Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.137847945Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.13858937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.138637076Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.138653595Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.138675563Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584} {"msg":"Crucible 8acfa87f-8213-4d72-aaec-55958796a780 has session id: 30489ae7-9522-4e90-b959-e75fbc9e2353","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139146372Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Upstairs opts: Upstairs UUID: 8acfa87f-8213-4d72-aaec-55958796a780, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139251653Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Crucible stats registered with UUID: 8acfa87f-8213-4d72-aaec-55958796a780","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139278441Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:35.139295919Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.13951673Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"io task","client":"2","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139557186Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"io task","client":"0","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139576524Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"io task","client":"1","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139717302Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"task":"crutest"} {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.13974946Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584} {"msg":"8acfa87f-8213-4d72-aaec-55958796a780 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139817203Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139838542Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"client":"0","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.13985679Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"client":"1","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139873969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"client":"2","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139890767Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"io task","client":"2","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139907866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"io task","client":"0","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.139925174Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"io task","client":"1","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"ds_connection connected from Ok(127.0.0.1:37870)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.140047674Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"io task","client":"2","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"ds_connection connected from Ok(127.0.0.1:63732)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.14009121Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"io task","client":"1","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"ds_connection connected from Ok(127.0.0.1:61747)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.140122447Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"io task","client":"0","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.141316253Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"client":"2","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.141369258Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"client":"0","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.141487908Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"client":"1","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.141886663Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.141931999Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"[0]R flush_numbers[0..12]: [159, 166, 162, 168, 167, 161, 161, 155, 159, 164, 167, 168]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.14204225Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"[0]R generation[0..12]: [90, 94, 92, 95, 94, 91, 91, 88, 90, 93, 94, 95]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142066427Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142087456Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"[1]R flush_numbers[0..12]: [159, 166, 162, 168, 167, 161, 161, 155, 159, 164, 167, 168]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142106464Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"[1]R generation[0..12]: [90, 94, 92, 95, 94, 91, 91, 88, 90, 93, 94, 95]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142124902Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142144681Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"[2]R flush_numbers[0..12]: [159, 166, 162, 168, 167, 161, 161, 155, 159, 164, 167, 168]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142163749Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"[2]R generation[0..12]: [90, 94, 92, 95, 94, 91, 91, 88, 90, 93, 94, 95]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142182397Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142202216Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142218714Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142244452Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Next flush: 169","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142260901Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142277139Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"":"downstairs","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142294178Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142309606Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"client":"0","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142326495Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"client":"1","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142343143Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"client":"2","":"downstairs","session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142360062Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142376211Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"8acfa87f-8213-4d72-aaec-55958796a780 is now active with session: 30489ae7-9522-4e90-b959-e75fbc9e2353","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142394059Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142409398Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584,"session_id":"30489ae7-9522-4e90-b959-e75fbc9e2353"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142425556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584} {"msg":"Activated sub_volume 8acfa87f-8213-4d72-aaec-55958796a780","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:35.142439585Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 351 len 32768 data: 6 9 7 10 11 9 8 8 02/30 Read block 555 len 12288 03/30 Read block 430 len 40960 04/30 Flush 05/30 Write block 150 len 28672 data: 7 7 6 7 5 6 7 06/30 Read block 453 len 16384 07/30 Read block 519 len 32768 08/30 Flush 09/30 Write block 275 len 8192 data: 11 12 10/30 Write block 353 len 12288 data: 8 11 12 11/30 Write block 460 len 20480 data: 7 8 8 8 9 12/30 Write block 596 len 4096 data: 7 13/30 Write block 117 len 28672 data: 6 6 5 5 6 6 8 14/30 Write block 528 len 16384 data: 6 6 10 9 15/30 Flush 16/30 Write block 265 len 28672 data: 7 7 8 7 8 8 8 17/30 Write block 590 len 28672 data: 6 7 8 7 7 8 8 18/30 Write block 94 len 8192 data: 8 10 19/30 Write block 516 len 36864 data: 6 5 4 5 5 7 8 7 8 20/30 Read block 415 len 8192 21/30 Write block 187 len 32768 data: 8 6 7 5 5 5 6 6 22/30 Read block 481 len 24576 23/30 Read block 9 len 16384 24/30 Read block 560 len 36864 25/30 Read block 98 len 4096 26/30 Write block 156 len 28672 data: 8 8 8 8 10 14 12 27/30 Read block 393 len 32768 28/30 Read block 83 len 36864 29/30 Write block 454 len 24576 data: 9 8 6 8 8 9 30/30 Write block 126 len 24576 data: 6 9 9 9 8 12 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 3 Done Sent Done false 1002 Acked Read 10 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 7 Done Sent Done false 1005 Acked Read 4 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 2 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Read 9 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 29 2 33 Done 28 1 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) None Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:35.161372945Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1584} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 F F F 1 020-039 94 94 94 166 166 166 F F F 2 040-059 92 92 92 162 162 162 F F F 3 060-079 95 95 95 168 168 168 F F F 4 080-099 94 94 94 167 167 167 T F T <--- 5 100-119 96 91 96 171 161 171 F F F <--- 6 120-139 96 91 96 171 161 171 T F T <--- 7 140-159 96 88 96 170 155 170 T F T <--- 8 160-179 90 90 90 159 159 159 T F T <--- 9 180-199 93 93 93 164 164 164 T F T <--- 10 200-219 94 94 94 167 167 167 F F F 11 220-239 95 95 95 168 168 168 F F F 12 240-259 94 94 94 167 167 167 F F F 13 260-279 96 94 96 171 166 171 T F T <--- 14 280-299 91 91 91 161 161 161 F F F 15 300-319 92 92 92 162 162 162 F F F 16 320-339 88 88 88 157 157 157 F F F 17 340-359 96 93 96 171 164 171 F T F <--- 18 360-379 95 95 95 168 168 168 F F F 19 380-399 95 95 95 168 168 168 F F F 20 400-419 95 95 95 168 168 168 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 92 92 92 163 163 163 T F T <--- 23 460-479 96 87 96 171 153 171 F F F <--- 24 480-499 93 93 93 164 164 164 F F F 25 500-519 90 90 90 159 159 159 T F T <--- 26 520-539 96 93 96 171 164 171 T F T <--- 27 540-559 94 94 94 167 167 167 F F F 28 560-579 88 88 88 157 157 157 F F F 29 580-599 96 94 96 171 167 171 T F T <--- Max gen: 96, Max flush: 171 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 F F F 1 020-039 94 94 94 166 166 166 F F F 2 040-059 92 92 92 162 162 162 F F F 3 060-079 95 95 95 168 168 168 F F F 4 080-099 97 97 97 172 172 172 F F F 5 100-119 96 96 96 171 171 171 F F F 6 120-139 97 97 97 172 172 172 F F F 7 140-159 97 97 97 172 172 172 F F F 8 160-179 97 97 97 172 172 172 F F F 9 180-199 97 97 97 172 172 172 F F F 10 200-219 94 94 94 167 167 167 F F F 11 220-239 95 95 95 168 168 168 F F F 12 240-259 94 94 94 167 167 167 F F F 13 260-279 97 97 97 172 172 172 F F F 14 280-299 91 91 91 161 161 161 F F F 15 300-319 92 92 92 162 162 162 F F F 16 320-339 88 88 88 157 157 157 F F F 17 340-359 96 96 96 171 171 171 F F F 18 360-379 95 95 95 168 168 168 F F F 19 380-399 95 95 95 168 168 168 F F F 20 400-419 95 95 95 168 168 168 F F F 21 420-439 83 83 83 147 147 147 F F F 22 440-459 97 97 97 172 172 172 F F F 23 460-479 96 96 96 171 171 171 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 97 97 97 172 172 172 F F F 26 520-539 97 97 97 172 172 172 F F F 27 540-559 94 94 94 167 167 167 F F F 28 560-579 88 88 88 157 157 157 F F F 29 580-599 97 97 97 172 172 172 F F F Max gen: 97, Max flush: 172 Begin loop 49 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.587262739Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.632857246Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.633693673Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.633743299Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.633763317Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.633785576Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592} {"msg":"Crucible 597bfe4c-7484-4088-a20b-b6d608b7cc39 has session id: b3edb02f-bc1a-4b03-bd22-2e6420a104d4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.634431969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Upstairs opts: Upstairs UUID: 597bfe4c-7484-4088-a20b-b6d608b7cc39, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.634490594Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Crucible stats registered with UUID: 597bfe4c-7484-4088-a20b-b6d608b7cc39","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.634521141Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:39.634546819Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.634803317Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"io task","client":"2","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.634855822Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"io task","client":"0","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.6348864Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"io task","client":"1","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.635042926Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"task":"crutest"} {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.635073103Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592} {"msg":"597bfe4c-7484-4088-a20b-b6d608b7cc39 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.635139877Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.635171655Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"client":"0","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.635204422Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"client":"1","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.635240789Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"client":"2","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.635272436Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"io task","client":"2","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.63533553Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"io task","client":"0","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.635381946Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"io task","client":"1","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"ds_connection connected from Ok(127.0.0.1:33943)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.635418603Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"io task","client":"2","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"ds_connection connected from Ok(127.0.0.1:61113)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.63545262Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"io task","client":"0","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"ds_connection connected from Ok(127.0.0.1:45130)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.635488547Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"io task","client":"1","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.636736678Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"client":"2","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.636907033Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"client":"1","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.63694947Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"client":"0","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637368373Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637480103Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"[0]R flush_numbers[0..12]: [159, 166, 162, 168, 172, 171, 172, 172, 172, 172, 167, 168]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637528339Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"[0]R generation[0..12]: [90, 94, 92, 95, 97, 96, 97, 97, 97, 97, 94, 95]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637551447Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637571935Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"[1]R flush_numbers[0..12]: [159, 166, 162, 168, 172, 171, 172, 172, 172, 172, 167, 168]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637594584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"[1]R generation[0..12]: [90, 94, 92, 95, 97, 96, 97, 97, 97, 97, 94, 95]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637613402Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.63763311Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"[2]R flush_numbers[0..12]: [159, 166, 162, 168, 172, 171, 172, 172, 172, 172, 167, 168]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637652328Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"[2]R generation[0..12]: [90, 94, 92, 95, 97, 96, 97, 97, 97, 97, 94, 95]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637674586Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637709814Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637734041Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637758029Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Next flush: 173","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637774598Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637791016Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"":"downstairs","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637808055Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637823724Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"client":"0","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637841192Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"client":"1","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.63785827Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"client":"2","":"downstairs","session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637878159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637894637Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"597bfe4c-7484-4088-a20b-b6d608b7cc39 is now active with session: b3edb02f-bc1a-4b03-bd22-2e6420a104d4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637912496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637931274Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592,"session_id":"b3edb02f-bc1a-4b03-bd22-2e6420a104d4"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637947913Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592} {"msg":"Activated sub_volume 597bfe4c-7484-4088-a20b-b6d608b7cc39","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:39.637962701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 182 len 28672 data: 8 8 5 6 7 9 7 02/30 Read block 436 len 40960 03/30 Write block 576 len 12288 data: 5 5 8 04/30 Write block 430 len 32768 data: 5 7 8 9 8 8 9 9 05/30 Read block 313 len 32768 06/30 Write block 33 len 36864 data: 8 5 5 5 5 5 7 8 6 07/30 Flush 08/30 Write block 189 len 36864 data: 8 6 6 6 7 7 7 11 12 09/30 Write block 333 len 16384 data: 6 6 6 5 10/30 Write block 155 len 8192 data: 7 9 11/30 Read block 84 len 4096 12/30 Read block 528 len 16384 13/30 Read block 231 len 32768 14/30 Read block 191 len 40960 15/30 Write block 231 len 12288 data: 9 10 7 16/30 Read block 392 len 28672 17/30 Write block 320 len 12288 data: 6 7 7 18/30 Read block 306 len 36864 19/30 Read block 129 len 8192 20/30 Read block 329 len 20480 21/30 Read block 139 len 24576 22/30 Read block 449 len 28672 23/30 Write block 115 len 32768 data: 4 4 7 7 6 6 7 7 24/30 Read block 110 len 40960 25/30 Write block 72 len 20480 data: 6 5 6 8 9 26/30 Flush 27/30 Write block 186 len 16384 data: 8 10 8 9 28/30 Write block 548 len 36864 data: 7 7 9 9 8 8 7 6 6 29/30 Write block 541 len 8192 data: 10 12 30/30 Read block 249 len 32768 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Sent Done false 1001 Acked Read 10 Done Sent Done false 1002 Acked Write 3 Done Sent Done false 1003 Acked Write 8 Done Sent Done false 1004 Acked Read 8 Done Sent Done false 1005 Acked Write 9 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 9 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 9 Done Sent Done false 1028 Acked Write 2 Done Sent Sent false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 2 32 Done 30 0 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:39.658071379Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1592} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 F F F 1 020-039 98 94 98 173 166 173 F F F <--- 2 040-059 98 92 98 173 162 173 F F F <--- 3 060-079 98 95 98 174 168 174 F F F <--- 4 080-099 97 97 97 172 172 172 F F F 5 100-119 98 96 98 174 171 174 F F F <--- 6 120-139 98 97 98 174 172 174 F F F <--- 7 140-159 98 97 98 174 172 174 F F F <--- 8 160-179 97 97 97 172 172 172 F F F 9 180-199 98 97 98 174 172 174 T F T <--- 10 200-219 94 94 94 167 167 167 F F F 11 220-239 98 95 98 174 168 174 F F F <--- 12 240-259 94 94 94 167 167 167 F F F 13 260-279 97 97 97 172 172 172 F F F 14 280-299 91 91 91 161 161 161 F F F 15 300-319 92 92 92 162 162 162 F F F 16 320-339 98 88 98 174 157 174 F F F <--- 17 340-359 96 96 96 171 171 171 F F F 18 360-379 95 95 95 168 168 168 F F F 19 380-399 95 95 95 168 168 168 F F F 20 400-419 95 95 95 168 168 168 F F F 21 420-439 98 83 98 173 147 173 F F F <--- 22 440-459 97 97 97 172 172 172 F F F 23 460-479 96 96 96 171 171 171 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 97 97 97 172 172 172 F F F 26 520-539 97 97 97 172 172 172 F F F 27 540-559 94 94 94 167 167 167 T F T <--- 28 560-579 98 88 98 173 157 173 F F F <--- 29 580-599 97 97 97 172 172 172 F F F Max gen: 98, Max flush: 174 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 F F F 1 020-039 98 98 98 173 173 173 F F F 2 040-059 98 98 98 173 173 173 F F F 3 060-079 98 98 98 174 174 174 F F F 4 080-099 97 97 97 172 172 172 F F F 5 100-119 98 98 98 174 174 174 F F F 6 120-139 98 98 98 174 174 174 F F F 7 140-159 98 98 98 174 174 174 F F F 8 160-179 97 97 97 172 172 172 F F F 9 180-199 99 99 99 175 175 175 F F F 10 200-219 94 94 94 167 167 167 F F F 11 220-239 98 98 98 174 174 174 F F F 12 240-259 94 94 94 167 167 167 F F F 13 260-279 97 97 97 172 172 172 F F F 14 280-299 91 91 91 161 161 161 F F F 15 300-319 92 92 92 162 162 162 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 96 96 96 171 171 171 F F F 18 360-379 95 95 95 168 168 168 F F F 19 380-399 95 95 95 168 168 168 F F F 20 400-419 95 95 95 168 168 168 F F F 21 420-439 98 98 98 173 173 173 F F F 22 440-459 97 97 97 172 172 172 F F F 23 460-479 96 96 96 171 171 171 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 97 97 97 172 172 172 F F F 26 520-539 97 97 97 172 172 172 F F F 27 540-559 99 99 99 175 175 175 F F F 28 560-579 98 98 98 173 173 173 F F F 29 580-599 97 97 97 172 172 172 F F F Max gen: 99, Max flush: 175 Begin loop 50 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.063633952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.108970611Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.109745554Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.109805049Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.109825407Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.109852915Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600} {"msg":"Crucible db170c2d-294a-42cf-a2a4-27c1b2802ebb has session id: b297d89f-6cc7-49c5-8a2c-fc462353c56f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.1103594Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Upstairs opts: Upstairs UUID: db170c2d-294a-42cf-a2a4-27c1b2802ebb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.110416375Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Crucible stats registered with UUID: db170c2d-294a-42cf-a2a4-27c1b2802ebb","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.110446743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:44.11047819Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.110688402Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"io task","client":"1","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.110731278Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"io task","client":"0","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.110750606Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"io task","client":"2","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.110912102Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"task":"crutest"} {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.11093628Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600} {"msg":"db170c2d-294a-42cf-a2a4-27c1b2802ebb active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.111021693Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.111044411Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"client":"0","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.111062609Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"client":"1","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.111079377Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"client":"2","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.111096056Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"io task","client":"2","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.111125704Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"io task","client":"0","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.111156871Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"io task","client":"1","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"ds_connection connected from Ok(127.0.0.1:64645)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.111425047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"io task","client":"2","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"ds_connection connected from Ok(127.0.0.1:51943)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.111472553Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"io task","client":"1","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"ds_connection connected from Ok(127.0.0.1:34740)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.1115055Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"io task","client":"0","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.112529371Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"client":"0","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.11265879Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"client":"2","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.112693577Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"client":"1","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113091072Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113209132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"[0]R flush_numbers[0..12]: [159, 173, 173, 174, 172, 174, 174, 174, 172, 175, 167, 174]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.11323682Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"[0]R generation[0..12]: [90, 98, 98, 98, 97, 98, 98, 98, 97, 99, 94, 98]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113257048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113277166Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"[1]R flush_numbers[0..12]: [159, 173, 173, 174, 172, 174, 174, 174, 172, 175, 167, 174]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113296604Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"[1]R generation[0..12]: [90, 98, 98, 98, 97, 98, 98, 98, 97, 99, 94, 98]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113314943Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113334281Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"[2]R flush_numbers[0..12]: [159, 173, 173, 174, 172, 174, 174, 174, 172, 175, 167, 174]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113353239Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"[2]R generation[0..12]: [90, 98, 98, 98, 97, 98, 98, 98, 97, 99, 94, 98]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113371608Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113392096Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113418004Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113448861Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Next flush: 176"Wait for a query_work_queue command to finish before sending IO ,"v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113473729Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113490227Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"":"downstairs","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113507496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113523075Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"client":"0","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113540403Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"client":"1","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113557532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"client":"2","":"downstairs","session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.1135749Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113591219Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"db170c2d-294a-42cf-a2a4-27c1b2802ebb is now active with session: b297d89f-6cc7-49c5-8a2c-fc462353c56f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113609147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113624726Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600,"session_id":"b297d89f-6cc7-49c5-8a2c-fc462353c56f"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113641124Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600} {"msg":"Activated sub_volume db170c2d-294a-42cf-a2a4-27c1b2802ebb","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:44.113655343Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 553 len 16384 data: 9 8 7 7 03/30 Write block 149 len 16384 data: 6 8 8 7 04/30 Read block 213 len 36864 05/30 Read block 534 len 8192 06/30 Read block 275 len 8192 07/30 Read block 271 len 28672 08/30 Write block 420 len 40960 data: 8 7 7 6 6 6 8 10 8 7 09/30 Write block 290 len 20480 data: 8 9 9 7 10 10/30 Flush 11/30 Write block 214 len 16384 data: 9 8 9 7 12/30 Write block 6 len 40960 data: 10 8 6 7 8 5 6 6 5 5 13/30 Write block 430 len 36864 data: 6 8 9 10 9 9 10 10 8 14/30 Read block 470 len 24576 15/30 Flush 16/30 Write block 365 len 20480 data: 8 8 7 4 5 17/30 Read block 165 len 16384 18/30 Write block 77 len 8192 data: 12 10 19/30 Read block 223 len 8192 20/30 Write block 373 len 32768 data: 3 4 5 6 7 7 7 7 21/30 Write block 528 len 36864 data: 7 7 11 10 9 9 8 7 11 22/30 Read block 112 len 36864 23/30 Read block 519 len 40960 24/30 Read block 244 len 20480 25/30 Write block 56 len 16384 data: 3 4 6 5 26/30 Read block 8 len 24576 27/30 Read block 324 len 28672 28/30 Write block 25 len 20480 data: 10 10 10 8 9 29/30 Read block 58 len 12288 30/30 Read block 341 len 16384 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Done Sent Done false 1002 Acked Write 4 Done Sent Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Read 2 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Read 7 Done Sent Done false 1007 Acked Write 10 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Write 9 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Read 10 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Read 7 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 1 30 Done 29 0 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1000)) Some(JobId(1014)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:44.13319808Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1600} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 100 90 100 178 159 178 F F F <--- 1 020-039 98 98 98 173 173 173 T F T <--- 2 040-059 98 98 98 173 173 173 T F T <--- 3 060-079 98 98 98 174 174 174 T F T <--- 4 080-099 97 97 97 172 172 172 F F F 5 100-119 98 98 98 174 174 174 F F F 6 120-139 98 98 98 174 174 174 F F F 7 140-159 100 98 100 177 174 177 F F F <--- 8 160-179 97 97 97 172 172 172 F F F 9 180-199 99 99 99 175 175 175 F F F 10 200-219 100 94 100 178 167 178 F F F <--- 11 220-239 98 98 98 174 174 174 F F F 12 240-259 94 94 94 167 167 167 F F F 13 260-279 97 97 97 172 172 172 F F F 14 280-299 100 91 100 177 161 177 F F F <--- 15 300-319 92 92 92 162 162 162 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 96 96 96 171 171 171 F F F 18 360-379 95 95 95 168 168 168 T F T <--- 19 380-399 95 95 95 168 168 168 T F T <--- 20 400-419 95 95 95 168 168 168 F F F 21 420-439 100 98 100 178 173 178 F F F <--- 22 440-459 97 97 97 172 172 172 F F F 23 460-479 96 96 96 171 171 171 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 97 97 97 172 172 172 F F F 26 520-539 97 97 97 172 172 172 T F T <--- 27 540-559 100 99 100 177 175 177 F F F <--- 28 560-579 98 98 98 173 173 173 F F F 29 580-599 97 97 97 172 172 172 F F F Max gen: 100, Max flush: 178 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 100 100 100 178 178 178 F F F 1 020-039 101 101 101 179 179 179 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 101 101 101 179 179 179 F F F 4 080-099 97 97 97 172 172 172 F F F 5 100-119 98 98 98 174 174 174 F F F 6 120-139 98 98 98 174 174 174 F F F 7 140-159 100 100 100 177 177 177 F F F 8 160-179 97 97 97 172 172 172 F F F 9 180-199 99 99 99 175 175 175 F F F 10 200-219 100 100 100 178 178 178 F F F 11 220-239 98 98 98 174 174 174 F F F 12 240-259 94 94 94 167 167 167 F F F 13 260-279 97 97 97 172 172 172 F F F 14 280-299 100 100 100 177 177 177 F F F 15 300-319 92 92 92 162 162 162 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 96 96 96 171 171 171 F F F 18 360-379 101 101 101 179 179 179 F F F 19 380-399 101 101 101 179 179 179 F F F 20 400-419 95 95 95 168 168 168 F F F 21 420-439 100 100 100 178 178 178 F F F 22 440-459 97 97 97 172 172 172 F F F 23 460-479 96 96 96 171 171 171 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 97 97 97 172 172 172 F F F 26 520-539 101 101 101 179 179 179 F F F 27 540-559 100 100 100 177 177 177 F F F 28 560-579 98 98 98 173 173 173 F F F 29 580-599 97 97 97 172 172 172 F F F Max gen: 101, Max flush: 179 Begin loop 51 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.560077957Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.604962796Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.605750017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.605800613Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.605817802Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.605840529Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608} {"msg":"Crucible b39fe2c5-4dbd-4e2f-acb9-3f0ef60bb7ad has session id: d2405392-411e-4df6-8840-f5c4b478a012","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.606310748Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Upstairs opts: Upstairs UUID: b39fe2c5-4dbd-4e2f-acb9-3f0ef60bb7ad, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.606378513Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Crucible stats registered with UUID: b39fe2c5-4dbd-4e2f-acb9-3f0ef60bb7ad","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.60640621Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:48.606423869Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.606616872Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"io task","client":"1","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.606669937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"io task","client":"0","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.606704694Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"io task","client":"2","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.606855091Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"task":"crutest"} {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.606895568Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608} {"msg":"b39fe2c5-4dbd-4e2f-acb9-3f0ef60bb7ad active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.606949773Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.60697943Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"client":"0","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.607017727Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"client":"1","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.607052464Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"client":"2","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.607085631Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"io task","client":"2","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.607120128Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"io task","client":"0","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.607150235Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"io task","client":"1","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"ds_connection connected from Ok(127.0.0.1:38937)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.607180593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"io task","client":"2","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"ds_connection connected from Ok(127.0.0.1:49257)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.60733311Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"io task","client":"1","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"ds_connection connected from Ok(127.0.0.1:50633)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.607378575Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"io task","client":"0","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.608442293Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"client":"0","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.608622427Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"client":"1","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.609110835Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.609149331Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.609372832Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"client":"2","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"[0]R flush_numbers[0..12]: [178, 179, 179, 179, 172, 174, 174, 177, 172, 175, 178, 174]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.609880548Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"[0]R generation[0..12]: [100, 101, 101, 101, 97, 98, 98, 100, 97, 99, 100, 98]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.609905755Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.609929593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"[1]R flush_numbers[0..12]: [178, 179, 179, 179, 172, 174, 174, 177, 172, 175, 178, 174]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.609952241Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"[1]R generation[0..12]: [100, 101, 101, 101, 97, 98, 98, 100, 97, 99, 100, 98]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.609983439Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610017976Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"[2]R flush_numbers[0..12]: [178, 179, 179, 179, 172, 174, 174, 177, 172, 175, 178, 174]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610040414Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"[2]R generation[0..12]: [100, 101, 101, 101, 97, 98, 98, 100, 97, 99, 100, 98]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610059542Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.61007949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610096309Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610113587Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Next flush: 180","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610130026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Wait for a query_work_queue command to finish before sending IO All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610151754Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"":"downstairs","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610168692Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610184201Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"client":"0","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.61020112Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"client":"1","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610220448Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"client":"2","":"downstairs","session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610237666Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610253925Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 b39fe2c5-4dbd-4e2f-acb9-3f0ef60bb7ad is now active with session: d2405392-411e-4df6-8840-f5c4b478a012","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610271883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610287432Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608,"session_id":"d2405392-411e-4df6-8840-f5c4b478a012"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610303631Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608} {"msg":"Activated sub_volume b39fe2c5-4dbd-4e2f-acb9-3f0ef60bb7ad","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:48.610317819Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 238 len 12288 data: 6 8 8 02/30 Write block 177 len 36864 data: 9 8 7 8 8 9 9 6 7 03/30 Write block 239 len 4096 data: 9 04/30 Read block 241 len 4096 05/30 Write block 269 len 24576 data: 9 9 9 12 13 13 06/30 Read block 373 len 16384 07/30 Write block 230 len 20480 data: 9 10 11 8 6 08/30 Read block 548 len 36864 09/30 Write block 395 len 20480 data: 6 9 9 8 8 10/30 Write block 163 len 12288 data: 11 9 8 11/30 Read block 507 len 24576 12/30 Read block 157 len 24576 13/30 Read block 520 len 12288 14/30 Read block 120 len 40960 15/30 Write block 304 len 8192 data: 9 10 16/30 Write block 13 len 40960 data: 7 6 6 6 5 7 9 6 8 7 17/30 Read block 107 len 28672 18/30 Read block 101 len 20480 19/30 Write block 118 len 32768 data: 8 7 7 8 8 9 7 7 20/30 Flush 21/30 Write block 449 len 32768 data: 12 14 12 10 8 10 9 7 22/30 Write block 537 len 32768 data: 11 10 8 10 11 13 10 8 23/30 Read block 481 len 36864 24/30 Read block 158 len 8192 25/30 Write block 148 len 12288 data: 6 7 9 26/30 Read block 254 len 8192 27/30 Write block 299 len 16384 data: 7 8 8 10 28/30 Write block 295 len 24576 data: 9 8 6 5 8 9 29/30 Write block 275 len 16384 data: 12 13 11 8 30/30 Read block 404 len 4096 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Read 1 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Read 4 Done Done Sent false 1006 Acked Write 5 Done Done Sent false 1007 Acked Read 9 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Write 2 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 8 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Read 2 Done Done Sent false 1024 Acked Write 3 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Write 6 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 1 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 27 28 Done 29 30 3 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:48.629063006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1608} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 100 180 180 178 F F F <--- 1 020-039 102 102 101 180 180 179 F F F <--- 2 040-059 101 101 101 179 179 179 F F F 3 060-079 101 101 101 179 179 179 F F F 4 080-099 97 97 97 172 172 172 F F F 5 100-119 102 102 98 180 180 174 F F F <--- 6 120-139 102 102 98 180 180 174 F F F <--- 7 140-159 100 100 100 177 177 177 T T F <--- 8 160-179 102 102 97 180 180 172 F F T <--- 9 180-199 102 102 99 180 180 175 F F T <--- 10 200-219 100 100 100 178 178 178 F F F 11 220-239 102 102 98 180 180 174 F F T <--- 12 240-259 102 102 94 180 180 167 F F T <--- 13 260-279 102 102 97 180 180 172 T T F <--- 14 280-299 100 100 100 177 177 177 T T F <--- 15 300-319 102 102 92 180 180 162 T T F <--- 16 320-339 98 98 98 174 174 174 F F F 17 340-359 96 96 96 171 171 171 F F F 18 360-379 101 101 101 179 179 179 F F F 19 380-399 102 102 101 180 180 179 F F F <--- 20 400-419 95 95 95 168 168 168 F F F 21 420-439 100 100 100 178 178 178 F F F 22 440-459 97 97 97 172 172 172 T T F <--- 23 460-479 96 96 96 171 171 171 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 97 97 97 172 172 172 F F F 26 520-539 101 101 101 179 179 179 T T F <--- 27 540-559 100 100 100 177 177 177 T T F <--- 28 560-579 98 98 98 173 173 173 F F F 29 580-599 97 97 97 172 172 172 F F F Max gen: 102, Max flush: 180 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 180 180 180 F F F 1 020-039 102 102 102 180 180 180 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 101 101 101 179 179 179 F F F 4 080-099 97 97 97 172 172 172 F F F 5 100-119 102 102 102 180 180 180 F F F 6 120-139 102 102 102 180 180 180 F F F 7 140-159 103 103 103 181 181 181 F F F 8 160-179 102 102 102 180 180 180 F F F 9 180-199 102 102 102 180 180 180 F F F 10 200-219 100 100 100 178 178 178 F F F 11 220-239 102 102 102 180 180 180 F F F 12 240-259 102 102 102 180 180 180 F F F 13 260-279 103 103 103 181 181 181 F F F 14 280-299 103 103 103 181 181 181 F F F 15 300-319 103 103 103 181 181 181 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 96 96 96 171 171 171 F F F 18 360-379 101 101 101 179 179 179 F F F 19 380-399 102 102 102 180 180 180 F F F 20 400-419 95 95 95 168 168 168 F F F 21 420-439 100 100 100 178 178 178 F F F 22 440-459 103 103 103 181 181 181 F F F 23 460-479 96 96 96 171 171 171 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 97 97 97 172 172 172 F F F 26 520-539 103 103 103 181 181 181 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 98 98 98 173 173 173 F F F 29 580-599 97 97 97 172 172 172 F F F Max gen: 103, Max flush: 181 Begin loop 52 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.055536568Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:53.098991181Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.099395226Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.130702688Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.131355701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.131402967Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.131419485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.131442243Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616} {"msg":"Crucible 1bd644c5-5d50-435d-b39e-3e23b883b4a7 has session id: e62d03e5-93ce-489b-a910-75809e89f25f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.131892544Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Upstairs opts: Upstairs UUID: 1bd644c5-5d50-435d-b39e-3e23b883b4a7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.131955468Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Crucible stats registered with UUID: 1bd644c5-5d50-435d-b39e-3e23b883b4a7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.131982506Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:53.132019463Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.132204997Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.132252673Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"io task","client":"0","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.1322859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"io task","client":"2","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.132450016Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"task":"crutest"} {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.132486592Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616} {"msg":"1bd644c5-5d50-435d-b39e-3e23b883b4a7 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.13251356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.132534228Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"client":"0","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.132556236Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"client":"1","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.132578804Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"client":"2","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.132600052Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"io task","client":"2","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.13262704Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"io task","client":"0","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.132665987Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"ds_connection connected from Ok(127.0.0.1:54924)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.132734421Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"io task","client":"0","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"ds_connection connected from Ok(127.0.0.1:35225)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.133016376Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"io task","client":"1","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"ds_connection connected from Ok(127.0.0.1:45877)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.133060582Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"io task","client":"2","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.133977272Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"client":"1","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134013159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"client":"2","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134099492Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"client":"0","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134525225Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134613107Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"[0]R flush_numbers[0..12]: [180, 180, 179, 179, 172, 180, 180, 181, 180, 180, 178, 180]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134708949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"[0]R generation[0..12]: [102, 102, 101, 101, 97, 102, 102, 103, 102, 102, 100, 102]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134732786Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134753305Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"[1]R flush_numbers[0..12]: [180, 180, 179, 179, 172, 180, 180, 181, 180, 180, 178, 180]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134772393Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"[1]R generation[0..12]: [102, 102, 101, 101, 97, 102, 102, 103, 102, 102, 100, 102]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134791292Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.13481107Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"[2]R flush_numbers[0..12]: [180, 180, 179, 179, 172, 180, 180, 181, 180, 180, 178, 180]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134834948Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"[2]R generation[0..12]: [102, 102, 101, 101, 97, 102, 102, 103, 102, 102, 100, 102]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134854126Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134873934Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Max found gen is 104","v":0,"name":"crucible"Wait for a query_work_queue command to finish before sending IO ,"level":30,"time":"2025-12-24T00:19:53.134896602Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Generation requested: 104 >= found:104","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134913991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Next flush: 182","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134930409Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134946678Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"":"downstairs","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134963677Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134979065Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"client":"0","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.134996243Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"client":"1","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.135013112Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"client":"2","":"downstairs","session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.135030321Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.135046489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"1bd644c5-5d50-435d-b39e-3e23b883b4a7 is now active with session: e62d03e5-93ce-489b-a910-75809e89f25f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.135064328Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.135082706Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616,"session_id":"e62d03e5-93ce-489b-a910-75809e89f25f"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.135100674Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616} {"msg":"Activated sub_volume 1bd644c5-5d50-435d-b39e-3e23b883b4a7","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:53.135114843Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 368 len 12288 data: 5 6 2 02/30 Read block 59 len 36864 03/30 Write block 280 len 20480 data: 8 9 8 8 8 04/30 Write block 93 len 12288 data: 7 9 11 05/30 Read block 212 len 36864 06/30 Flush 07/30 Write block 298 len 24576 data: 6 9 10 9 11 10 08/30 Read block 131 len 24576 09/30 Write block 160 len 36864 data: 11 15 13 12 10 9 9 6 6 10/30 Write block 1 len 28672 data: 2 3 4 5 9 11 9 11/30 Write block 449 len 32768 data: 13 15 13 11 9 11 10 8 12/30 Write block 506 len 4096 data: 6 13/30 Write block 150 len 8192 data: 10 9 14/30 Write block 428 len 20480 data: 9 8 7 9 10 15/30 Read block 296 len 8192 16/30 Read block 105 len 32768 17/30 Read block 254 len 16384 18/30 Read block 527 len 16384 19/30 Flush 20/30 Write block 188 len 28672 data: 9 10 7 7 7 8 8 21/30 Write block 84 len 12288 data: 11 9 9 22/30 Read block 379 len 40960 23/30 Read block 49 len 16384 24/30 Write block 66 len 40960 data: 9 8 9 8 7 6 7 6 7 9 25/30 Write block 401 len 28672 data: 7 7 8 5 7 8 9 26/30 Read block 174 len 40960 27/30 Read block 292 len 36864 28/30 Write block 217 len 40960 data: 8 9 8 8 10 9 6 8 9 7 29/30 Write block 563 len 28672 data: 3 4 4 5 5 6 6 30/30 Write block 385 len 8192 data: 8 7 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:24 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1006 Acked Write 6 Done Done Done false 1007 Acked Read 6 Done Done Done false 1008 Acked Write 9 Done Done Done false 1009 Acked Write 7 Done Done Done false 1010 Acked Write 8 Done Done Done false 1011 Acked Write 1 Done Done Done false 1012 Acked Write 2 Done Done Done false 1013 Acked Write 5 Done Done Done false 1014 Acked Read 2 Done Done Done false 1015 Acked Read 8 Done Done Done false 1016 Acked Read 4 Done Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Read 10 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 13 3 3 19 Done 11 21 21 53 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1005)) Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: 1005 1004 1003 1002 1001 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:53.154119327Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1616} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 104 104 180 183 183 T F F <--- 1 020-039 102 102 102 180 180 180 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 101 101 101 179 179 179 F T T <--- 4 080-099 104 104 104 182 182 182 F T T <--- 5 100-119 102 102 102 180 180 180 F F F 6 120-139 102 102 102 180 180 180 F F F 7 140-159 103 104 104 181 183 183 T F F <--- 8 160-179 102 104 104 180 183 183 T F F <--- 9 180-199 102 102 102 180 180 180 F T T <--- 10 200-219 100 100 100 178 178 178 F T T <--- 11 220-239 102 102 102 180 180 180 F T T <--- 12 240-259 102 102 102 180 180 180 F F F 13 260-279 103 103 103 181 181 181 F F F 14 280-299 104 104 104 182 183 183 T F F <--- 15 300-319 103 104 104 181 183 183 T F F <--- 16 320-339 98 98 98 174 174 174 F F F 17 340-359 96 96 96 171 171 171 F F F 18 360-379 104 104 104 182 182 182 F F F 19 380-399 102 102 102 180 180 180 F T T <--- 20 400-419 95 95 95 168 168 168 F T T <--- 21 420-439 100 104 104 178 183 183 T F F <--- 22 440-459 103 104 104 181 183 183 T F F <--- 23 460-479 96 96 96 171 171 171 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 97 104 104 172 183 183 T F F <--- 26 520-539 103 103 103 181 181 181 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 98 98 98 173 173 173 F T T <--- 29 580-599 97 97 97 172 172 172 F F F Max gen: 104, Max flush: 183 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 102 102 102 180 180 180 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 105 105 105 184 184 184 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 102 102 102 180 180 180 F F F 6 120-139 102 102 102 180 180 180 F F F 7 140-159 104 104 104 183 183 183 F F F 8 160-179 104 104 104 183 183 183 F F F 9 180-199 105 105 105 184 184 184 F F F 10 200-219 105 105 105 184 184 184 F F F 11 220-239 105 105 105 184 184 184 F F F 12 240-259 102 102 102 180 180 180 F F F 13 260-279 103 103 103 181 181 181 F F F 14 280-299 104 104 104 183 183 183 F F F 15 300-319 104 104 104 183 183 183 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 96 96 96 171 171 171 F F F 18 360-379 104 104 104 182 182 182 F F F 19 380-399 105 105 105 184 184 184 F F F 20 400-419 105 105 105 184 184 184 F F F 21 420-439 104 104 104 183 183 183 F F F 22 440-459 104 104 104 183 183 183 F F F 23 460-479 96 96 96 171 171 171 F F F 24 480-499 93 93 93 164 164 164 F F F 25 500-519 104 104 104 183 183 183 F F F 26 520-539 103 103 103 181 181 181 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 105 105 105 184 184 184 F F F 29 580-599 97 97 97 172 172 172 F F F Max gen: 105, Max flush: 184 Begin loop 53 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.61639124Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.66115427Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.661995316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.662049152Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.66206746Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.662089498Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624} {"msg":"Crucible 9adb5c80-ae88-4a07-a0eb-b39f066e5922 has session id: dd147381-3eba-4c86-8849-8ea724034138","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.662607533Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Upstairs opts: Upstairs UUID: 9adb5c80-ae88-4a07-a0eb-b39f066e5922, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.662657068Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Crucible stats registered with UUID: 9adb5c80-ae88-4a07-a0eb-b39f066e5922","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.662679327Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:57.662696385Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.662968831Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"io task","client":"1","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663007708Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"io task","client":"0","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663034556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"io task","client":"2","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663219099Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"task":"crutest"} {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663242848Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624} {"msg":"9adb5c80-ae88-4a07-a0eb-b39f066e5922 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.66332391Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663343989Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"client":"0","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663362337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"client":"1","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663379356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"client":"2","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663396204Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"io task","client":"2","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663413493Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"io task","client":"0","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663462478Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"io task","client":"1","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"ds_connection connected from Ok(127.0.0.1:35001)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663616235Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"io task","client":"0","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"ds_connection connected from Ok(127.0.0.1:51845)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663661971Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"io task","client":"2","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"ds_connection connected from Ok(127.0.0.1:42635)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.663699298Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"io task","client":"1","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.664904703Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"client":"2","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.664983246Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"client":"0","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.665532698Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.665574624Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.666532331Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"client":"1","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"[0]R flush_numbers[0..12]: [183, 180, 179, 184, 184, 180, 180, 183, 183, 184, 184, 184]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667018339Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"[0]R generation[0..12]: [104, 102, 101, 105, 105, 102, 102, 104, 104, 105, 105, 105]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667048386Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667069044Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"[1]R flush_numbers[0..12]: [183, 180, 179, 184, 184, 180, 180, 183, 183, 184, 184, 184]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667088253Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"[1]R generation[0..12]: [104, 102, 101, 105, 105, 102, 102, 104, 104, 105, 105, 105]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667107301Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667127399Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"[2]R flush_numbers[0..12]: [183, 180, 179, 184, 184, 180, 180, 183, 183, 184, 184, 184]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667146237Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"[2]R generation[0..12]: [104, 102, 101, 105, 105, 102, 102, 104, 104, 105, 105, 105]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667165426Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667184874Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Max found gen is 106","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667201372Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667218441Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Next flush: 185","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.66723468Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667250968Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"":"downstairs","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667271637Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667287425Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"client":"0","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:19:57.667304634Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"client":"1","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667327661Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"client":"2","":"downstairs","session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.66734469Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667361169Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"9adb5c80-ae88-4a07-a0eb-b39f066e5922 is now active with session: dd147381-3eba-4c86-8849-8ea724034138","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667379057Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667394276Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624,"session_id":"dd147381-3eba-4c86-8849-8ea724034138"} {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667410184Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624} {"msg":"Activated sub_volume 9adb5c80-ae88-4a07-a0eb-b39f066e5922","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:19:57.667424173Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 154 len 36864 data: 6 8 10 9 9 9 12 16 14 02/30 Write block 496 len 12288 data: 9 6 4 03/30 Write block 307 len 28672 data: 10 8 7 10 9 8 7 04/30 Write block 250 len 32768 data: 10 10 9 7 6 8 9 10 05/30 Read block 524 len 16384 06/30 Read block 358 len 28672 07/30 Write block 583 len 20480 data: 4 4 5 5 7 08/30 Write block 404 len 32768 data: 6 8 9 10 10 9 9 10 09/30 Read block 369 len 12288 10/30 Read block 445 len 28672 11/30 Write block 143 len 28672 data: 8 8 8 5 5 7 8 12/30 Read block 388 len 4096 13/30 Read block 106 len 12288 14/30 Write block 353 len 8192 data: 9 12 15/30 Read block 320 len 8192 16/30 Read block 426 len 36864 17/30 Read block 338 len 40960 18/30 Read block 52 len 12288 19/30 Write block 380 len 8192 data: 8 8 20/30 Write block 569 len 8192 data: 7 7 21/30 Flush 22/30 Write block 589 len 12288 data: 6 7 8 23/30 Write block 456 len 4096 data: 9 24/30 Read block 251 len 16384 25/30 Write block 369 len 40960 data: 7 3 3 4 4 5 6 7 8 8 26/30 Flush 27/30 Write block 148 len 40960 data: 8 9 11 10 8 8 7 9 11 10 28/30 Write block 471 len 24576 data: 10 8 7 8 8 5 29/30 Write block 519 len 16384 data: 6 6 8 9 30/30 Read block 410 len 4096 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Write 7 Done Done Done false 1003 Acked Write 8 Done Sent Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Read 7 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Write 8 Done Sent Done false 1008 Acked Read 3 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Read 1 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Write 2 Done Sent Done false 1014 Acked Read 2 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Write 2 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Write 6 Done Sent Sent false 1028 Acked Write 4 Done Sent Sent false 1029 Acked Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 3 30 Done 30 3 27 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:19:57.686536198Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1624} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 102 102 102 180 180 180 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 105 105 105 184 184 184 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 102 102 102 180 180 180 F F F 6 120-139 102 102 102 180 180 180 F F F 7 140-159 106 104 106 185 183 185 T T T <--- 8 160-179 106 104 106 185 183 185 F T F <--- 9 180-199 105 105 105 184 184 184 F F F 10 200-219 105 105 105 184 184 184 F F F 11 220-239 105 105 105 184 184 184 F F F 12 240-259 106 102 106 185 180 185 F F F <--- 13 260-279 103 103 103 181 181 181 F F F 14 280-299 104 104 104 183 183 183 F F F 15 300-319 106 104 106 185 183 185 F T F <--- 16 320-339 98 98 98 174 174 174 F F F 17 340-359 106 96 106 185 171 185 F F F <--- 18 360-379 106 104 106 186 182 186 F F F <--- 19 380-399 106 105 106 185 184 185 F F F <--- 20 400-419 106 105 106 185 184 185 F F F <--- 21 420-439 104 104 104 183 183 183 F F F 22 440-459 106 104 106 186 183 186 F F F <--- 23 460-479 96 96 96 171 171 171 T F T <--- 24 480-499 106 93 106 185 164 185 F T F <--- 25 500-519 104 104 104 183 183 183 T F T <--- 26 520-539 103 103 103 181 181 181 T F T <--- 27 540-559 103 103 103 181 181 181 F F F 28 560-579 106 105 106 185 184 185 F F F <--- 29 580-599 106 97 106 186 172 186 F F F <--- Max gen: 106, Max flush: 186 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 102 102 102 180 180 180 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 105 105 105 184 184 184 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 102 102 102 180 180 180 F F F 6 120-139 102 102 102 180 180 180 F F F 7 140-159 107 107 107 187 187 187 F F F 8 160-179 106 106 106 185 185 185 F F F 9 180-199 105 105 105 184 184 184 F F F 10 200-219 105 105 105 184 184 184 F F F 11 220-239 105 105 105 184 184 184 F F F 12 240-259 106 106 106 185 185 185 F F F 13 260-279 103 103 103 181 181 181 F F F 14 280-299 104 104 104 183 183 183 F F F 15 300-319 106 106 106 185 185 185 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 106 106 106 185 185 185 F F F 18 360-379 106 106 106 186 186 186 F F F 19 380-399 106 106 106 185 185 185 F F F 20 400-419 106 106 106 185 185 185 F F F 21 420-439 104 104 104 183 183 183 F F F 22 440-459 106 106 106 186 186 186 F F F 23 460-479 107 107 107 187 187 187 F F F 24 480-499 106 106 106 185 185 185 F F F 25 500-519 107 107 107 187 187 187 F F F 26 520-539 107 107 107 187 187 187 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 106 106 106 185 185 185 F F F 29 580-599 106 106 106 186 186 186 F F F Max gen: 107, Max flush: 187 Begin loop 54 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.110130081Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:02.153669587Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.154070022Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.185111617Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.185827475Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.18588442Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.185908818Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.185937275Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632} {"msg":"Crucible a7dca14e-8c82-4470-9294-4accdbca7341 has session id: 3dc662ec-acd1-434b-9f31-7aaa707e0cfa","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.186518235Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Upstairs opts: Upstairs UUID: a7dca14e-8c82-4470-9294-4accdbca7341, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.186591208Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Crucible stats registered with UUID: a7dca14e-8c82-4470-9294-4accdbca7341","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.186630005Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:02.186652713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.186828037Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"io task","client":"0","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.186873724Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"io task","client":"2","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.186897382Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"io task","client":"1","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187048548Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"task":"crutest"} {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187073436Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632} {"msg":"a7dca14e-8c82-4470-9294-4accdbca7341 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187135161Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187155399Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"client":"0","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187173007Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"client":"1","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187191356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"client":"2","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187212604Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"io task","client":"2","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187236192Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"io task","client":"0","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187268769Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"io task","client":"1","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"ds_connection connected from Ok(127.0.0.1:54370)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187313195Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"io task","client":"1","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"ds_connection connected from Ok(127.0.0.1:42151)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187345943Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"io task","client":"2","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"ds_connection connected from Ok(127.0.0.1:42708)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.187379009Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"io task","client":"0","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.18863381Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"client":"1","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.188696405Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"client":"0","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.188774358Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"client":"2","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189222529Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189308991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"[0]R flush_numbers[0..12]: [183, 180, 179, 184, 184, 180, 180, 187, 185, 184, 184, 184]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189373126Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"[0]R generation[0..12]: [104, 102, 101, 105, 105, 102, 102, 107, 106, 105, 105, 105]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189397244Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189417762Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"[1]R flush_numbers[0..12]: [183, 180, 179, 184, 184, 180, 180, 187, 185, 184, 184, 184]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.18944036Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"[1]R generation[0..12]: [104, 102, 101, 105, 105, 102, 102, 107, 106, 105, 105, 105]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189459898Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189479897Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"[2]R flush_numbers[0..12]: [183, 180, 179, 184, 184, 180, 180, 187, 185, 184, 184, 184]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189498805Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"[2]R generation[0..12]: [104, 102, 101, 105, 105, 102, 102, 107, 106, 105, 105, 105]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189517673Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189537221Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.18955357Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189570649Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Wait for a query_work_queue command to finish before sending IO Next flush: 188","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189592877Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189609215Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"":"downstairs","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189626374Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189641632Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"client":"0","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189658721Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"client":"1","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.18967552Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"client":"2","":"downstairs","session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189692698Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189710856Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"a7dca14e-8c82-4470-9294-4accdbca7341 is now active with session: 3dc662ec-acd1-434b-9f31-7aaa707e0cfa","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189729305Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189744654Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632,"session_id":"3dc662ec-acd1-434b-9f31-7aaa707e0cfa"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189761572Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632} {"msg":"Activated sub_volume a7dca14e-8c82-4470-9294-4accdbca7341","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:02.189776451Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 164 len 32768 data: 11 10 10 7 7 7 10 10 02/30 Write block 219 len 16384 data: 9 9 11 10 03/30 Read block 170 len 12288 04/30 Read block 479 len 4096 05/30 Read block 35 len 12288 06/30 Read block 260 len 32768 07/30 Read block 9 len 28672 08/30 Read block 114 len 8192 09/30 Write block 414 len 32768 data: 10 9 9 5 6 7 9 8 10/30 Write block 110 len 20480 data: 7 7 5 6 6 11/30 Read block 88 len 4096 12/30 Read block 194 len 40960 13/30 Read block 264 len 12288 14/30 Read block 174 len 40960 15/30 Flush 16/30 Write block 491 len 36864 data: 8 8 8 8 9 10 7 5 6 17/30 Read block 229 len 28672 18/30 Flush 19/30 Write block 67 len 36864 data: 9 10 9 8 7 8 7 8 10 20/30 Flush 21/30 Write block 480 len 24576 data: 7 7 9 10 11 8 22/30 Write block 448 len 24576 data: 11 14 16 14 12 10 23/30 Write block 563 len 28672 data: 4 5 5 6 6 7 8 24/30 Flush 25/30 Write block 166 len 12288 data: 11 8 8 26/30 Read block 551 len 4096 27/30 Read block 323 len 28672 28/30 Read block 4 len 8192 29/30 Read block 489 len 8192 30/30 Read block 311 len 40960 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 4 Sent Done Done false 1002 Acked Read 3 Sent Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Read 3 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Read 7 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Read 10 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:02.2105564Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1632} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 102 102 102 180 180 180 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 105 108 108 184 190 190 F F F <--- 4 080-099 105 105 105 184 184 184 F F F 5 100-119 102 108 108 180 188 188 F F F <--- 6 120-139 102 102 102 180 180 180 F F F 7 140-159 107 107 107 187 187 187 F F F 8 160-179 106 108 108 185 188 188 T T T <--- 9 180-199 105 105 105 184 184 184 F F F 10 200-219 105 108 108 184 188 188 F F F <--- 11 220-239 105 108 108 184 188 188 F F F <--- 12 240-259 106 106 106 185 185 185 F F F 13 260-279 103 103 103 181 181 181 F F F 14 280-299 104 104 104 183 183 183 F F F 15 300-319 106 106 106 185 185 185 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 106 106 106 185 185 185 F F F 18 360-379 106 106 106 186 186 186 F F F 19 380-399 106 106 106 185 185 185 F F F 20 400-419 106 108 108 185 188 188 F F F <--- 21 420-439 104 108 108 183 188 188 F F F <--- 22 440-459 106 108 108 186 191 191 F F F <--- 23 460-479 107 107 107 187 187 187 F F F 24 480-499 106 108 108 185 191 191 F F F <--- 25 500-519 107 107 107 187 187 187 F F F 26 520-539 107 107 107 187 187 187 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 106 108 108 185 191 191 F F F <--- 29 580-599 106 106 106 186 186 186 F F F Max gen: 108, Max flush: 191 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 102 102 102 180 180 180 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 108 108 108 190 190 190 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 108 108 108 188 188 188 F F F 6 120-139 102 102 102 180 180 180 F F F 7 140-159 107 107 107 187 187 187 F F F 8 160-179 109 109 109 192 192 192 F F F 9 180-199 105 105 105 184 184 184 F F F 10 200-219 108 108 108 188 188 188 F F F 11 220-239 108 108 108 188 188 188 F F F 12 240-259 106 106 106 185 185 185 F F F 13 260-279 103 103 103 181 181 181 F F F 14 280-299 104 104 104 183 183 183 F F F 15 300-319 106 106 106 185 185 185 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 106 106 106 185 185 185 F F F 18 360-379 106 106 106 186 186 186 F F F 19 380-399 106 106 106 185 185 185 F F F 20 400-419 108 108 108 188 188 188 F F F 21 420-439 108 108 108 188 188 188 F F F 22 440-459 108 108 108 191 191 191 F F F 23 460-479 107 107 107 187 187 187 F F F 24 480-499 108 108 108 191 191 191 F F F 25 500-519 107 107 107 187 187 187 F F F 26 520-539 107 107 107 187 187 187 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 108 108 108 191 191 191 F F F 29 580-599 106 106 106 186 186 186 F F F Max gen: 109, Max flush: 192 Begin loop 55 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.635770433Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.681505787Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.68227155Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.682329465Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.682350274Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.682377261Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640} {"msg":"Crucible 0dfdac47-4e3e-4d2c-9b88-7e00d3d28033 has session id: beddc1ab-9120-49d0-b040-15a461e08547","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.682819243Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Upstairs opts: Upstairs UUID: 0dfdac47-4e3e-4d2c-9b88-7e00d3d28033, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.682875838Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Crucible stats registered with UUID: 0dfdac47-4e3e-4d2c-9b88-7e00d3d28033","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.682906905Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:06.682934223Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683176591Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"io task","client":"1","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683217898Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"io task","client":"2","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683238876Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"io task","client":"0","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683481525Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"task":"crutest"} {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683527761Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640} {"msg":"0dfdac47-4e3e-4d2c-9b88-7e00d3d28033 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683554509Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683575227Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"client":"0","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683597465Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"client":"1","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683621763Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"client":"2","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.68365164Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"io task","client":"2","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683679148Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"io task","client":"0","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683708275Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"io task","client":"1","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"ds_connection connected from Ok(127.0.0.1:45922)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683757171Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"io task","client":"0","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"ds_connection connected from Ok(127.0.0.1:37478)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683796137Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"io task","client":"2","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"ds_connection connected from Ok(127.0.0.1:38642)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.683900419Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"io task","client":"1","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685182047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"client":"1","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685220253Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"client":"0","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685248361Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"client":"2","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685799403Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.68582973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"[0]R flush_numbers[0..12]: [183, 180, 179, 190, 184, 188, 180, 187, 192, 184, 188, 188]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685854648Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"[0]R generation[0..12]: [104, 102, 101, 108, 105, 108, 102, 107, 109, 105, 108, 108]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685874587Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685894665Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"[1]R flush_numbers[0..12]: [183, 180, 179, 190, 184, 188, 180, 187, 192, 184, 188, 188]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685913813Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"[1]R generation[0..12]: [104, 102, 101, 108, 105, 108, 102, 107, 109, 105, 108, 108]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685932931Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.68595285Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"[2]R flush_numbers[0..12]: [183, 180, 179, 190, 184, 188, 180, 187, 192, 184, 188, 188]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685971868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"[2]R generation[0..12]: [104, 102, 101, 108, 105, 108, 102, 107, 109, 105, 108, 108]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.685990846Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686010205Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686026863Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686047191Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Next flush: 193","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.68606617Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686094137Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"":"downstairs","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2025-12-24T00:20:06.686124865Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686140883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"client":"0","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686158282Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"client":"1","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.68617986Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"client":"2","":"downstairs","session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686208077Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686226276Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"0dfdac47-4e3e-4d2c-9b88-7e00d3d28033 is now active with session: beddc1ab-9120-49d0-b040-15a461e08547","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686244284Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686259643Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640,"session_id":"beddc1ab-9120-49d0-b040-15a461e08547"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686282321Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640} {"msg":"Activated sub_volume 0dfdac47-4e3e-4d2c-9b88-7e00d3d28033","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:06.686305669Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 151 len 12288 data: 11 9 9 02/30 Read block 222 len 4096 03/30 Read block 162 len 36864 04/30 Write block 508 len 24576 data: 7 8 7 8 8 8 05/30 Write block 22 len 36864 data: 8 9 10 11 11 11 9 10 9 06/30 Read block 115 len 28672 07/30 Flush 08/30 Write block 135 len 8192 data: 11 8 09/30 Read block 526 len 16384 10/30 Read block 276 len 36864 11/30 Write block 327 len 32768 data: 7 8 8 6 8 7 7 7 12/30 Read block 87 len 16384 13/30 Write block 353 len 40960 data: 10 13 13 10 9 9 9 10 11 11 14/30 Read block 269 len 4096 15/30 Flush 16/30 Flush 17/30 Write block 166 len 8192 data: 12 9 18/30 Read block 474 len 32768 19/30 Write block 414 len 28672 data: 11 10 10 6 7 8 10 20/30 Read block 190 len 28672 21/30 Read block 367 len 24576 22/30 Flush 23/30 Write block 482 len 40960 data: 10 11 12 9 9 10 7 6 6 9 24/30 Write block 500 len 28672 data: 6 7 5 6 7 7 7 25/30 Read block 310 len 4096 26/30 Write block 187 len 40960 data: 11 10 11 8 8 8 9 9 8 12 27/30 Write block 368 len 36864 data: 6 8 4 4 5 5 6 7 8 28/30 Write block 271 len 16384 data: 10 13 14 14 29/30 Read block 460 len 12288 30/30 Read block 323 len 36864 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Read 9 Done Done Sent false 1003 Acked Write 6 Done Done Sent false 1004 Acked Write 9 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 2 Done Done Sent false 1008 Acked Read 4 Done Done Sent false 1009 Acked Read 9 Done Done Sent false 1010 Acked Write 8 Done Done Sent false 1011 Acked Read 4 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Write 7 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 28 29 Done 29 30 2 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:06.707145483Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1640} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 110 110 102 193 193 180 F F F <--- 2 040-059 101 101 101 179 179 179 F F F 3 060-079 108 108 108 190 190 190 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 108 108 108 188 188 188 F F F 6 120-139 110 110 102 194 194 180 F F F <--- 7 140-159 110 110 107 193 193 187 F F T <--- 8 160-179 110 110 109 196 196 192 F F F <--- 9 180-199 105 105 105 184 184 184 T T F <--- 10 200-219 108 108 108 188 188 188 F F F 11 220-239 108 108 108 188 188 188 F F F 12 240-259 106 106 106 185 185 185 F F F 13 260-279 103 103 103 181 181 181 T T F <--- 14 280-299 104 104 104 183 183 183 F F F 15 300-319 106 106 106 185 185 185 F F F 16 320-339 110 110 98 194 194 174 F F F <--- 17 340-359 110 110 106 194 194 185 F F F <--- 18 360-379 110 110 106 194 194 186 T T F <--- 19 380-399 106 106 106 185 185 185 F F F 20 400-419 110 110 108 196 196 188 F F F <--- 21 420-439 110 110 108 196 196 188 F F F <--- 22 440-459 108 108 108 191 191 191 F F F 23 460-479 107 107 107 187 187 187 F F F 24 480-499 108 108 108 191 191 191 T T F <--- 25 500-519 110 110 107 193 193 187 T T F <--- 26 520-539 107 107 107 187 187 187 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 108 108 108 191 191 191 F F F 29 580-599 106 106 106 186 186 186 F F F Max gen: 110, Max flush: 196 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 110 110 110 193 193 193 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 108 108 108 190 190 190 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 108 108 108 188 188 188 F F F 6 120-139 110 110 110 194 194 194 F F F 7 140-159 110 110 110 193 193 193 F F F 8 160-179 110 110 110 196 196 196 F F F 9 180-199 111 111 111 197 197 197 F F F 10 200-219 108 108 108 188 188 188 F F F 11 220-239 108 108 108 188 188 188 F F F 12 240-259 106 106 106 185 185 185 F F F 13 260-279 111 111 111 197 197 197 F F F 14 280-299 104 104 104 183 183 183 F F F 15 300-319 106 106 106 185 185 185 F F F 16 320-339 110 110 110 194 194 194 F F F 17 340-359 110 110 110 194 194 194 F F F 18 360-379 111 111 111 197 197 197 F F F 19 380-399 106 106 106 185 185 185 F F F 20 400-419 110 110 110 196 196 196 F F F 21 420-439 110 110 110 196 196 196 F F F 22 440-459 108 108 108 191 191 191 F F F 23 460-479 107 107 107 187 187 187 F F F 24 480-499 111 111 111 197 197 197 F F F 25 500-519 111 111 111 197 197 197 F F F 26 520-539 107 107 107 187 187 187 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 108 108 108 191 191 191 F F F 29 580-599 106 106 106 186 186 186 F F F Max gen: 111, Max flush: 197 Begin loop 56 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.113936831Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.158538584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.159247492Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.159298247Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.159322816Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.159352013Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648} {"msg":"Crucible 70e5d4d6-c717-4627-845e-7233b6d8e455 has session id: 4418b01c-332f-4bdd-8de6-2b24e85cd65e","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.159852239Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Upstairs opts: Upstairs UUID: 70e5d4d6-c717-4627-845e-7233b6d8e455, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.159924313Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Crucible stats registered with UUID: 70e5d4d6-c717-4627-845e-7233b6d8e455","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.15996304Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:11.159993657Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160149104Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"io task","client":"0","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.16019131Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"io task","client":"1","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160211458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"io task","client":"2","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160403171Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"task":"crutest"} {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.16042356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648} {"msg":"70e5d4d6-c717-4627-845e-7233b6d8e455 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160505582Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160525821Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"client":"0","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160543439Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"client":"1","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160568007Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"client":"2","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160599144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"io task","client":"2","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160677158Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"io task","client":"0","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160722943Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"io task","client":"1","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"ds_connection connected from Ok(127.0.0.1:33234)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.16076048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"io task","client":"0","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"ds_connection connected from Ok(127.0.0.1:42491)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160884279Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"io task","client":"1","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"ds_connection connected from Ok(127.0.0.1:62534)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.160928905Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"io task","client":"2","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.161958256Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"client":"1","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.161992963Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"client":"0","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.162567323Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.162650286Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.163865Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"client":"2","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"[0]R flush_numbers[0..12]: [183, 193, 179, 190, 184, 188, 194, 193, 196, 197, 188, 188]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164420531Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"[0]R generation[0..12]: [104, 110, 101, 108, 105, 108, 110, 110, 110, 111, 108, 108]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164463248Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164494225Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"[1]R flush_numbers[0..12]: [183, 193, 179, 190, 184, 188, 194, 193, 196, 197, 188, 188]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164519273Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"[1]R generation[0..12]: [104, 110, 101, 108, 105, 108, 110, 110, 110, 111, 108, 108]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164543561Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164567908Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"[2]R flush_numbers[0..12]: [183, 193, 179, 190, 184, 188, 194, 193, 196, 197, 188, 188]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164591756Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"[2]R generation[0..12]: [104, 110, 101, 108, 105, 108, 110, 110, 110, 111, 108, 108]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164623214Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.16467059Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Max found gen is 112","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164694207Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Generation requested: 112 >= found:112","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164726554Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Next flush: 198","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164765231Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164802508Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"":"downstairs","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 time":"2025-12-24T00:20:11.164842295Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164882411Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"client":"0","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164910909Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"client":"1","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164932527Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"client":"2","":"downstairs","session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164957045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164977303Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"70e5d4d6-c717-4627-845e-7233b6d8e455 is now active with session: 4418b01c-332f-4bdd-8de6-2b24e85cd65e","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.164999791Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.165018949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648,"session_id":"4418b01c-332f-4bdd-8de6-2b24e85cd65e"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.165039147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648} {"msg":"Activated sub_volume 70e5d4d6-c717-4627-845e-7233b6d8e455","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:11.165056626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 583 len 32768 data: 5 5 6 6 8 6 7 8 02/30 Write block 109 len 40960 data: 8 8 8 6 7 7 5 5 8 9 03/30 Read block 3 len 16384 04/30 Write block 335 len 32768 data: 7 6 7 6 6 8 7 6 05/30 Write block 286 len 24576 data: 8 7 7 5 9 10 06/30 Write block 441 len 12288 data: 10 10 10 07/30 Read block 514 len 16384 08/30 Read block 372 len 20480 09/30 Write block 297 len 24576 data: 7 7 10 11 10 12 10/30 Read block 170 len 24576 11/30 Read block 14 len 4096 12/30 Write block 567 len 40960 data: 7 8 9 8 7 6 7 7 6 6 13/30 Read block 25 len 28672 14/30 Read block 46 len 28672 15/30 Flush 16/30 Write block 215 len 24576 data: 9 10 9 10 10 10 17/30 Write block 433 len 36864 data: 11 10 10 11 11 9 8 9 11 18/30 Write block 335 len 40960 data: 8 7 8 7 7 9 8 7 6 6 19/30 Write block 510 len 20480 data: 8 9 9 9 7 20/30 Write block 593 len 20480 data: 8 8 9 9 7 21/30 Flush 22/30 Flush 23/30 Write block 112 len 8192 data: 7 8 24/30 Flush 25/30 Write block 573 len 8192 data: 8 8 26/30 Read block 29 len 28672 27/30 Write block 262 len 24576 data: 12 10 8 8 8 9 28/30 Write block 100 len 8192 data: 9 10 29/30 Read block 580 len 28672 30/30 Read block 261 len 24576 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Sent false 1001 Acked Write 10 Done Done Sent false 1002 Acked Read 4 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Write 6 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Write 5 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 30 31 Done 30 29 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:11.184760379Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1648} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 110 110 110 193 193 193 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 108 108 108 190 190 190 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 112 112 108 201 201 188 T T F <--- 6 120-139 110 110 110 194 194 194 F F F 7 140-159 110 110 110 193 193 193 F F F 8 160-179 110 110 110 196 196 196 F F F 9 180-199 111 111 111 197 197 197 F F F 10 200-219 112 112 108 199 199 188 F F F <--- 11 220-239 112 112 108 199 199 188 F F F <--- 12 240-259 106 106 106 185 185 185 F F F 13 260-279 111 111 111 197 197 197 T T F <--- 14 280-299 112 112 104 198 198 183 F F F <--- 15 300-319 112 112 106 198 198 185 F F F <--- 16 320-339 112 112 110 199 199 194 F F F <--- 17 340-359 112 112 110 199 199 194 F F F <--- 18 360-379 111 111 111 197 197 197 F F F 19 380-399 106 106 106 185 185 185 F F F 20 400-419 110 110 110 196 196 196 F F F 21 420-439 112 112 110 199 199 196 F F F <--- 22 440-459 112 112 108 199 199 191 F F F <--- 23 460-479 107 107 107 187 187 187 F F F 24 480-499 111 111 111 197 197 197 F F F 25 500-519 112 112 111 199 199 197 F F F <--- 26 520-539 107 107 107 187 187 187 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 112 112 108 198 198 191 T T F <--- 29 580-599 112 112 106 199 199 186 F F F <--- Max gen: 112, Max flush: 201 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 110 110 110 193 193 193 F F F 2 040-059 101 101 101 179 179 179 F F F 3 060-079 108 108 108 190 190 190 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 113 113 113 202 202 202 F F F 6 120-139 110 110 110 194 194 194 F F F 7 140-159 110 110 110 193 193 193 F F F 8 160-179 110 110 110 196 196 196 F F F 9 180-199 111 111 111 197 197 197 F F F 10 200-219 112 112 112 199 199 199 F F F 11 220-239 112 112 112 199 199 199 F F F 12 240-259 106 106 106 185 185 185 F F F 13 260-279 113 113 113 202 202 202 F F F 14 280-299 112 112 112 198 198 198 F F F 15 300-319 112 112 112 198 198 198 F F F 16 320-339 112 112 112 199 199 199 F F F 17 340-359 112 112 112 199 199 199 F F F 18 360-379 111 111 111 197 197 197 F F F 19 380-399 106 106 106 185 185 185 F F F 20 400-419 110 110 110 196 196 196 F F F 21 420-439 112 112 112 199 199 199 F F F 22 440-459 112 112 112 199 199 199 F F F 23 460-479 107 107 107 187 187 187 F F F 24 480-499 111 111 111 197 197 197 F F F 25 500-519 112 112 112 199 199 199 F F F 26 520-539 107 107 107 187 187 187 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 113 113 113 202 202 202 F F F 29 580-599 112 112 112 199 199 199 F F F Max gen: 113, Max flush: 202 Begin loop 57 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.592834945Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:15.63615949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.636583553Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.667298816Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.66806215Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.668109485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.668125584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.668147752Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656} {"msg":"Crucible 54ec98f5-6327-418d-ac53-c3d9d55495af has session id: 07622050-67cf-4d8c-b82f-ec0a0a6af4c6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.668692084Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Upstairs opts: Upstairs UUID: 54ec98f5-6327-418d-ac53-c3d9d55495af, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.668753259Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Crucible stats registered with UUID: 54ec98f5-6327-418d-ac53-c3d9d55495af","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.668780067Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:15.668798285Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.66897532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"io task","client":"1","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669010367Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"io task","client":"2","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669029705Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"io task","client":"0","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669186532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"task":"crutest"} {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669211609Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656} {"msg":"54ec98f5-6327-418d-ac53-c3d9d55495af active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669272144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669293102Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"client":"0","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669310661Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"client":"1","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669327539Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"client":"2","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669344338Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"io task","client":"2","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.66943202Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"io task","client":"0","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669570928Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"io task","client":"1","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"ds_connection connected from Ok(127.0.0.1:51251)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669619474Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"io task","client":"2","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"ds_connection connected from Ok(127.0.0.1:37330)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.66966152Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"io task","client":"1","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"ds_connection connected from Ok(127.0.0.1:55431)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.669692967Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"io task","client":"0","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.670771883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"client":"2","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.67080745Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"client":"1","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.670957327Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"client":"0","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671410058Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671451884Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"[0]R flush_numbers[0..12]: [183, 193, 179, 190, 184, 202, 194, 193, 196, 197, 199, 199]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671513909Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"[0]R generation[0..12]: [104, 110, 101, 108, 105, 113, 110, 110, 110, 111, 112, 112]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671547436Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671568964Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"[1]R flush_numbers[0..12]: [183, 193, 179, 190, 184, 202, 194, 193, 196, 197, 199, 199]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671588222Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"[1]R generation[0..12]: [104, 110, 101, 108, 105, 113, 110, 110, 110, 111, 112, 112]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.67160723Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671627039Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"[2]R flush_numbers[0..12]: [183, 193, 179, 190, 184, 202, 194, 193, 196, 197, 199, 199]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671650337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"[2]R generation[0..12]: [104, 110, 101, 108, 105, 113, 110, 110, 110, 111, 112, 112]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671669655Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671689433Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671705982Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671723211Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Next flush: 203","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671739319Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671755438Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"":"downstairs","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671772726Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671788175Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"client":"0","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671805393Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"client":"1","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Wait for a query_work_queue command to finish before sending IO Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671827771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"client":"2","":"downstairs","session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.67184496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671861088Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"54ec98f5-6327-418d-ac53-c3d9d55495af is now active with session: 07622050-67cf-4d8c-b82f-ec0a0a6af4c6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671879077Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671894356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656,"session_id":"07622050-67cf-4d8c-b82f-ec0a0a6af4c6"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671910514Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656} {"msg":"Activated sub_volume 54ec98f5-6327-418d-ac53-c3d9d55495af","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:15.671926473Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 264 len 32768 data: 9 9 9 10 8 10 10 11 02/30 Write block 364 len 40960 data: 10 9 9 8 7 9 5 5 6 6 03/30 Write block 380 len 12288 data: 9 9 7 04/30 Write block 241 len 32768 data: 8 8 9 6 6 9 10 9 05/30 Read block 243 len 8192 06/30 Write block 46 len 16384 data: 7 6 5 6 07/30 Write block 446 len 28672 data: 11 13 12 15 17 15 13 08/30 Read block 264 len 16384 09/30 Read block 322 len 16384 10/30 Write block 382 len 16384 data: 8 7 12 9 11/30 Read block 189 len 16384 12/30 Read block 269 len 4096 13/30 Write block 492 len 4096 data: 9 14/30 Write block 277 len 12288 data: 12 9 7 15/30 Read block 260 len 36864 16/30 Write block 357 len 24576 data: 10 10 10 11 12 12 17/30 Write block 589 len 36864 data: 8 9 9 9 9 9 10 10 8 18/30 Read block 225 len 24576 19/30 Write block 209 len 24576 data: 7 6 7 7 8 10 20/30 Write block 529 len 32768 data: 8 12 11 10 10 9 8 12 21/30 Read block 90 len 32768 22/30 Read block 186 len 28672 23/30 Write block 192 len 36864 data: 9 10 10 9 13 13 15 15 13 24/30 Write block 590 len 8192 data: 10 10 25/30 Read block 248 len 8192 26/30 Write block 509 len 28672 data: 9 9 10 10 10 8 5 27/30 Write block 569 len 36864 data: 10 9 8 7 9 9 7 7 6 28/30 Read block 550 len 28672 29/30 Read block 418 len 24576 30/30 Read block 297 len 16384 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 10 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Read 2 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Write 3 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Write 7 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Read 4 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:15.690078571Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1656} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 110 110 110 193 193 193 F F F 2 040-059 101 101 101 179 179 179 F T T <--- 3 060-079 108 108 108 190 190 190 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 113 113 113 202 202 202 F F F 6 120-139 110 110 110 194 194 194 F F F 7 140-159 110 110 110 193 193 193 F F F 8 160-179 110 110 110 196 196 196 F F F 9 180-199 111 111 111 197 197 197 F T T <--- 10 200-219 112 112 112 199 199 199 F T T <--- 11 220-239 112 112 112 199 199 199 F F F 12 240-259 106 106 106 185 185 185 F T T <--- 13 260-279 113 113 113 202 202 202 T T T 14 280-299 112 112 112 198 198 198 F F F 15 300-319 112 112 112 198 198 198 F F F 16 320-339 112 112 112 199 199 199 F F F 17 340-359 112 112 112 199 199 199 F T T <--- 18 360-379 111 111 111 197 197 197 F T T <--- 19 380-399 106 106 106 185 185 185 F T T <--- 20 400-419 110 110 110 196 196 196 F F F 21 420-439 112 112 112 199 199 199 F F F 22 440-459 112 112 112 199 199 199 F T T <--- 23 460-479 107 107 107 187 187 187 F F F 24 480-499 111 111 111 197 197 197 F T T <--- 25 500-519 112 112 112 199 199 199 F T T <--- 26 520-539 107 107 107 187 187 187 F T T <--- 27 540-559 103 103 103 181 181 181 F F F 28 560-579 113 113 113 202 202 202 F T T <--- 29 580-599 112 112 112 199 199 199 F T T <--- Max gen: 113, Max flush: 202 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 57 Verifying data now Files /tmp/test_live_repair/test_repair_verify.data and /tmp/test_live_repair/test_repair_verify.data.last differ 280c280,282 < 11, --- > 12, > 9, > 7, 282,283d283 < 6, < 8, diff found after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 110 110 110 193 193 193 F F F 2 040-059 114 114 114 203 203 203 F F F 3 060-079 108 108 108 190 190 190 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 113 113 113 202 202 202 F F F 6 120-139 110 110 110 194 194 194 F F F 7 140-159 110 110 110 193 193 193 F F F 8 160-179 110 110 110 196 196 196 F F F 9 180-199 114 114 114 203 203 203 F F F 10 200-219 114 114 114 203 203 203 F F F 11 220-239 112 112 112 199 199 199 F F F 12 240-259 114 114 114 203 203 203 F F F 13 260-279 114 114 114 203 203 203 F F F 14 280-299 112 112 112 198 198 198 F F F 15 300-319 112 112 112 198 198 198 F F F 16 320-339 112 112 112 199 199 199 F F F 17 340-359 114 114 114 203 203 203 F F F 18 360-379 114 114 114 203 203 203 F F F 19 380-399 114 114 114 203 203 203 F F F 20 400-419 110 110 110 196 196 196 F F F 21 420-439 112 112 112 199 199 199 F F F 22 440-459 114 114 114 203 203 203 F F F 23 460-479 107 107 107 187 187 187 F F F 24 480-499 114 114 114 203 203 203 F F F 25 500-519 114 114 114 203 203 203 F F F 26 520-539 114 114 114 203 203 203 F F F 27 540-559 103 103 103 181 181 181 F F F 28 560-579 114 114 114 203 203 203 F F F 29 580-599 114 114 114 203 203 203 F F F Max gen: 114, Max flush: 203 Begin loop 58 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.150172852Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:20.198367832Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.198806024Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.23224455Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.233020083Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.233067758Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.233084067Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.233106095Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667} {"msg":"Crucible 013c4079-8142-4382-85e7-80b7bb89538f has session id: 83032b22-229d-497b-8dcc-e7f25e7af512","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.233642878Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Upstairs opts: Upstairs UUID: 013c4079-8142-4382-85e7-80b7bb89538f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.233701593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Crucible stats registered with UUID: 013c4079-8142-4382-85e7-80b7bb89538f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.233731661Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:20.233749719Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.233937252Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"io task","client":"0","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.233981909Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"io task","client":"2","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234014926Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"io task","client":"1","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234180211Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"task":"crutest"} {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234209989Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667} {"msg":"013c4079-8142-4382-85e7-80b7bb89538f active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234269883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234291122Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"client":"0","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.2343094Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"client":"1","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234331918Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"client":"2","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234386833Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"io task","client":"2","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234441489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"io task","client":"0","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234478895Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"io task","client":"1","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"ds_connection connected from Ok(127.0.0.1:57500)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234512422Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"io task","client":"0","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"ds_connection connected from Ok(127.0.0.1:62565)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234595505Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"io task","client":"2","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"ds_connection connected from Ok(127.0.0.1:61747)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.234677928Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"io task","client":"1","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.235811339Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"client":"1","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.235848466Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"client":"2","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236104874Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"client":"0","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236401528Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236482641Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"[0]R flush_numbers[0..12]: [183, 193, 203, 190, 184, 202, 194, 193, 196, 203, 203, 199]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236617609Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"[0]R generation[0..12]: [104, 110, 114, 108, 105, 113, 110, 110, 110, 114, 114, 112]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236641657Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236661735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"[1]R flush_numbers[0..12]: [183, 193, 203, 190, 184, 202, 194, 193, 196, 203, 203, 199]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236681064Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"[1]R generation[0..12]: [104, 110, 114, 108, 105, 113, 110, 110, 110, 114, 114, 112]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236699992Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.23671983Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"[2]R flush_numbers[0..12]: [183, 193, 203, 190, 184, 202, 194, 193, 196, 203, 203, 199]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236739228Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"[2]R generation[0..12]: [104, 110, 114, 108, 105, 113, 110, 110, 110, 114, 114, 112]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236758197Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236778095Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Max found gen is 115","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236794843Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Generation requested: 116 >= found:115","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236821021Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Next flush: 204","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.23683726Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236853558Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"":"downstairs","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236870907Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236886475Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"client":"0","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236903494Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"client":"1","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236920612Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"client":"2","":"downstairs","session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236937991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236957959Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"013c4079-8142-4382-85e7-80b7bb89538f is now active with session: 83032b22-229d-497b-8dcc-e7f25e7af512","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236976308Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.236991966Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667,"session_id":"83032b22-229d-497b-8dcc-e7f25e7af512"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.237008235Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667} {"msg":"Activated sub_volume 013c4079-8142-4382-85e7-80b7bb89538f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:20.237022414Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 462 len 20480 data: 9 9 10 10 8 02/30 Write block 183 len 12288 data: 10 7 8 03/30 Read block 512 len 8192 04/30 Write block 339 len 8192 data: 8 10 05/30 Write block 134 len 20480 data: 9 12 9 9 10 06/30 Read block 200 len 32768 07/30 Read block 76 len 4096 08/30 Write block 550 len 4096 data: 10 09/30 Write block 477 len 40960 data: 3 3 4 8 8 11 12 13 10 10 10/30 Write block 355 len 16384 data: 14 11 11 11 11/30 Write block 309 len 36864 data: 8 11 10 9 8 8 7 6 6 12/30 Flush 13/30 Flush 14/30 Write block 68 len 16384 data: 11 10 9 8 15/30 Write block 32 len 28672 data: 6 9 6 6 6 6 6 16/30 Flush 17/30 Write block 250 len 4096 data: 11 18/30 Write block 197 len 28672 data: 14 16 16 14 14 11 10 19/30 Read block 495 len 40960 20/30 Read block 43 len 20480 21/30 Read block 4 len 28672 22/30 Read block 488 len 24576 23/30 Read block 259 len 36864 24/30 Write block 361 len 36864 data: 13 13 10 11 10 10 9 8 10 25/30 Read block 543 len 24576 26/30 Read block 117 len 28672 27/30 Read block 534 len 20480 28/30 Read block 363 len 32768 29/30 Write block 396 len 24576 data: 10 10 9 9 7 8 30/30 Read block 575 len 24576 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 3 Sent Done Done false 1002 Acked Read 2 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Write 6 Sent Done Done false 1029 Acked Read 6 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:20.256383507Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1667} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 110 116 116 193 206 206 F F F <--- 2 040-059 114 114 114 203 203 203 F F F 3 060-079 108 116 116 190 206 206 F F F <--- 4 080-099 105 105 105 184 184 184 F F F 5 100-119 113 113 113 202 202 202 F F F 6 120-139 110 116 116 194 204 204 F F F <--- 7 140-159 110 110 110 193 193 193 F F F 8 160-179 110 110 110 196 196 196 F F F 9 180-199 114 116 116 203 204 204 F T T <--- 10 200-219 114 114 114 203 203 203 F T T <--- 11 220-239 112 112 112 199 199 199 F F F 12 240-259 114 114 114 203 203 203 F T T <--- 13 260-279 114 114 114 203 203 203 F F F 14 280-299 112 112 112 198 198 198 F F F 15 300-319 112 116 116 198 204 204 F F F <--- 16 320-339 112 116 116 199 204 204 F F F <--- 17 340-359 114 116 116 203 204 204 F F F <--- 18 360-379 114 114 114 203 203 203 F T T <--- 19 380-399 114 114 114 203 203 203 F T T <--- 20 400-419 110 110 110 196 196 196 F T T <--- 21 420-439 112 112 112 199 199 199 F F F 22 440-459 114 114 114 203 203 203 F F F 23 460-479 107 116 116 187 204 204 T F F <--- 24 480-499 114 116 116 203 204 204 F F F <--- 25 500-519 114 114 114 203 203 203 F F F 26 520-539 114 114 114 203 203 203 F F F 27 540-559 103 116 116 181 204 204 F F F <--- 28 560-579 114 114 114 203 203 203 F F F 29 580-599 114 114 114 203 203 203 F F F Max gen: 116, Max flush: 206 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 58 Verifying data now No change after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F F F 1 020-039 116 116 116 206 206 206 F F F 2 040-059 114 114 114 203 203 203 F F F 3 060-079 116 116 116 206 206 206 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 113 113 113 202 202 202 F F F 6 120-139 116 116 116 204 204 204 F F F 7 140-159 110 110 110 193 193 193 F F F 8 160-179 110 110 110 196 196 196 F F F 9 180-199 117 117 117 207 207 207 F F F 10 200-219 117 117 117 207 207 207 F F F 11 220-239 112 112 112 199 199 199 F F F 12 240-259 117 117 117 207 207 207 F F F 13 260-279 114 114 114 203 203 203 F F F 14 280-299 112 112 112 198 198 198 F F F 15 300-319 116 116 116 204 204 204 F F F 16 320-339 116 116 116 204 204 204 F F F 17 340-359 116 116 116 204 204 204 F F F 18 360-379 117 117 117 207 207 207 F F F 19 380-399 117 117 117 207 207 207 F F F 20 400-419 117 117 117 207 207 207 F F F 21 420-439 112 112 112 199 199 199 F F F 22 440-459 114 114 114 203 203 203 F F F 23 460-479 116 116 116 204 204 204 F F F 24 480-499 116 116 116 204 204 204 F F F 25 500-519 114 114 114 203 203 203 F F F 26 520-539 114 114 114 203 203 203 F F F 27 540-559 116 116 116 204 204 204 F F F 28 560-579 114 114 114 203 203 203 F F F 29 580-599 114 114 114 203 203 203 F F F Max gen: 117, Max flush: 207 Begin loop 59 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.703151582Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:24.74712355Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.747529594Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.778650312Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.77936764Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.779411756Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.779428495Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.779450423Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675} {"msg":"Crucible ac868f18-c2bf-46d2-ad45-988afccc60fd has session id: 1c4f9934-8199-4e48-9e6a-7f895e10d939","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780000025Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Upstairs opts: Upstairs UUID: ac868f18-c2bf-46d2-ad45-988afccc60fd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.78005066Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Crucible stats registered with UUID: ac868f18-c2bf-46d2-ad45-988afccc60fd","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780071459Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:24.780088167Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780305048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"io task","client":"2","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780349214Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"io task","client":"0","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780381732Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"io task","client":"1","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780495811Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"task":"crutest"} {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.78051551Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675} {"msg":"ac868f18-c2bf-46d2-ad45-988afccc60fd active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780584934Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780609702Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"client":"0","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.78062754Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"client":"1","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780644369Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"client":"2","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780662977Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"io task","client":"2","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780693484Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"io task","client":"0","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780724252Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"io task","client":"1","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"ds_connection connected from Ok(127.0.0.1:42294)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780754939Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"io task","client":"0","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"ds_connection connected from Ok(127.0.0.1:45398)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780887368Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"io task","client":"1","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"ds_connection connected from Ok(127.0.0.1:55734)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.780932073Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"io task","client":"2","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782060755Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"client":"1","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.78211155Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"client":"2","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782138698Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"client":"0","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782651343Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782753245Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"[0]R flush_numbers[0..12]: [183, 206, 203, 206, 184, 202, 204, 193, 196, 207, 207, 199]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782832367Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"[0]R generation[0..12]: [104, 116, 114, 116, 105, 113, 116, 110, 110, 117, 117, 112]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782862835Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782883473Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"[1]R flush_numbers[0..12]: [183, 206, 203, 206, 184, 202, 204, 193, 196, 207, 207, 199]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782903101Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"[1]R generation[0..12]: [104, 116, 114, 116, 105, 113, 116, 110, 110, 117, 117, 112]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.78292259Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782942288Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"[2]R flush_numbers[0..12]: [183, 206, 203, 206, 184, 202, 204, 193, 196, 207, 207, 199]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782966036Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"[2]R generation[0..12]: [104, 116, 114, 116, 105, 113, 116, 110, 110, 117, 117, 112]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.782998973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.78303288Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":Wait for a query_work_queue command to finish before sending IO 1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Max found gen is 118","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783069807Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Generation requested: 118 >= found:118","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783103194Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Next flush: 208","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783135531Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783168348Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"":"downstairs","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783201286Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783231413Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"client":"0","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783270599Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"client":"1","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783293727Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"client":"2","":"downstairs","session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783311496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783328024Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"ac868f18-c2bf-46d2-ad45-988afccc60fd is now active with session: 1c4f9934-8199-4e48-9e6a-7f895e10d939","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783346243Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783361841Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675,"session_id":"1c4f9934-8199-4e48-9e6a-7f895e10d939"} {"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.78337814Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675} {"msg":"Activated sub_volume ac868f18-c2bf-46d2-ad45-988afccc60fd","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:24.783392229Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 72 len 24576 data: 9 8 9 11 10 13 03/30 Write block 252 len 28672 data: 10 8 7 9 10 11 8 04/30 Write block 569 len 8192 data: 11 10 05/30 Write block 312 len 4096 data: 10 06/30 Write block 294 len 8192 data: 11 10 07/30 Write block 160 len 32768 data: 13 17 15 13 12 11 13 10 08/30 Read block 561 len 20480 09/30 Read block 274 len 20480 10/30 Write block 10 len 16384 data: 9 6 7 8 11/30 Write block 303 len 16384 data: 11 10 11 10 12/30 Write block 189 len 20480 data: 12 9 9 10 11 13/30 Write block 257 len 28672 data: 12 9 9 10 11 13 11 14/30 Write block 203 len 40960 data: 11 9 8 9 9 8 8 7 8 8 15/30 Write block 57 len 28672 data: 5 7 6 5 5 7 7 16/30 Write block 589 len 32768 data: 9 11 11 10 10 10 11 11 17/30 Write block 22 len 12288 data: 9 10 11 18/30 Read block 375 len 8192 19/30 Write block 228 len 4096 data: 9 20/30 Read block 538 len 24576 21/30 Read block 254 len 12288 22/30 Read block 510 len 40960 23/30 Write block 136 len 28672 data: 10 10 11 9 10 11 11 24/30 Write block 303 len 20480 data: 12 11 12 11 11 25/30 Read block 18 len 16384 26/30 Write block 209 len 12288 data: 9 8 9 27/30 Read block 586 len 40960 28/30 Read block 21 len 24576 29/30 Read block 474 len 16384 30/30 Write block 373 len 36864 data: 7 7 8 9 9 9 8 10 10 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 6 Done Done Done false 1002 Acked Write 7 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Read 5 Sent Done Done false 1008 Acked Read 5 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Read 2 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Read 10 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 1 30 Done 1 28 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:24.799567879Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1675} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 183 183 183 F T T <--- 1 020-039 116 116 116 206 206 206 F T T <--- 2 040-059 114 114 114 203 203 203 F T T <--- 3 060-079 116 116 116 206 206 206 T T T 4 080-099 105 105 105 184 184 184 F F F 5 100-119 113 113 113 202 202 202 F F F 6 120-139 116 116 116 204 204 204 F T T <--- 7 140-159 110 110 110 193 193 193 F T T <--- 8 160-179 110 110 110 196 196 196 F T T <--- 9 180-199 117 117 117 207 207 207 F T T <--- 10 200-219 117 117 117 207 207 207 F T T <--- 11 220-239 112 112 112 199 199 199 F T T <--- 12 240-259 117 117 117 207 207 207 F T T <--- 13 260-279 114 114 114 203 203 203 F T T <--- 14 280-299 112 112 112 198 198 198 F T T <--- 15 300-319 116 116 116 204 204 204 F T T <--- 16 320-339 116 116 116 204 204 204 F F F 17 340-359 116 116 116 204 204 204 F F F 18 360-379 117 117 117 207 207 207 F T T <--- 19 380-399 117 117 117 207 207 207 F T T <--- 20 400-419 117 117 117 207 207 207 F F F 21 420-439 112 112 112 199 199 199 F F F 22 440-459 114 114 114 203 203 203 F F F 23 460-479 116 116 116 204 204 204 F F F 24 480-499 116 116 116 204 204 204 F F F 25 500-519 114 114 114 203 203 203 F F F 26 520-539 114 114 114 203 203 203 F F F 27 540-559 116 116 116 204 204 204 F F F 28 560-579 114 114 114 203 203 203 F T T <--- 29 580-599 114 114 114 203 203 203 F T T <--- Max gen: 117, Max flush: 207 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 59 Verifying data now Files /tmp/test_live_repair/test_repair_verify.data and /tmp/test_live_repair/test_repair_verify.data.last differ 63,66c63,66 < 4, < 4, < 6, < 6, --- > 5, > 5, > 7, > 7, diff found after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 208 208 208 F F F 1 020-039 118 118 118 208 208 208 F F F 2 040-059 118 118 118 208 208 208 F F F 3 060-079 118 118 118 208 208 208 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 113 113 113 202 202 202 F F F 6 120-139 118 118 118 208 208 208 F F F 7 140-159 118 118 118 208 208 208 F F F 8 160-179 118 118 118 208 208 208 F F F 9 180-199 118 118 118 208 208 208 F F F 10 200-219 118 118 118 208 208 208 F F F 11 220-239 118 118 118 208 208 208 F F F 12 240-259 118 118 118 208 208 208 F F F 13 260-279 118 118 118 208 208 208 F F F 14 280-299 118 118 118 208 208 208 F F F 15 300-319 118 118 118 208 208 208 F F F 16 320-339 116 116 116 204 204 204 F F F 17 340-359 116 116 116 204 204 204 F F F 18 360-379 118 118 118 208 208 208 F F F 19 380-399 118 118 118 208 208 208 F F F 20 400-419 117 117 117 207 207 207 F F F 21 420-439 112 112 112 199 199 199 F F F 22 440-459 114 114 114 203 203 203 F F F 23 460-479 116 116 116 204 204 204 F F F 24 480-499 116 116 116 204 204 204 F F F 25 500-519 114 114 114 203 203 203 F F F 26 520-539 114 114 114 203 203 203 F F F 27 540-559 116 116 116 204 204 204 F F F 28 560-579 118 118 118 208 208 208 F F F 29 580-599 118 118 118 208 208 208 F F F Max gen: 118, Max flush: 208 Begin loop 60 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.290488495Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.335523701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.336344189Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.336400734Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.336427392Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.336472228Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686} {"msg":"Crucible c87dc9d6-f2b1-4000-aae4-0cfd55f7c7f9 has session id: bf306c22-d4b8-4f81-a9bb-c5ea47267dab","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.336953396Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Upstairs opts: Upstairs UUID: c87dc9d6-f2b1-4000-aae4-0cfd55f7c7f9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.337014311Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Crucible stats registered with UUID: c87dc9d6-f2b1-4000-aae4-0cfd55f7c7f9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.337048188Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:29.337073786Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.337329813Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.33737218Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"io task","client":"0","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.337393418Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"io task","client":"2","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.337564553Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"task":"crutest"} {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.33759189Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686} {"msg":"c87dc9d6-f2b1-4000-aae4-0cfd55f7c7f9 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.337650205Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.337681583Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"client":"0","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.33771375Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"client":"1","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.337866137Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"client":"2","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.337905743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"io task","client":"2","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.33794243Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"io task","client":"0","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.337977317Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"ds_connection connected from Ok(127.0.0.1:40860)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.338018883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"ds_connection connected from Ok(127.0.0.1:47330)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.33805617Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"io task","client":"2","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"ds_connection connected from Ok(127.0.0.1:38715)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.338087477Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"io task","client":"0","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.339294482Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"client":"0","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.339515583Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"client":"2","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.339800718Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"client":"1","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.339965714Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340062995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"[0]R flush_numbers[0..12]: [208, 208, 208, 208, 184, 202, 208, 208, 208, 208, 208, 208]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340303024Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"[0]R generation[0..12]: [118, 118, 118, 118, 105, 113, 118, 118, 118, 118, 118, 118]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340329282Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.3403497Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"[1]R flush_numbers[0..12]: [208, 208, 208, 208, 184, 202, 208, 208, 208, 208, 208, 208]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340369099Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"[1]R generation[0..12]: [118, 118, 118, 118, 105, 113, 118, 118, 118, 118, 118, 118]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340387717Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340407115Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"[2]R flush_numbers[0..12]: [208, 208, 208, 208, 184, 202, 208, 208, 208, 208, 208, 208]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340429423Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"[2]R generation[0..12]: [118, 118, 118, 118, 105, 113, 118, 118, 118, 118, 118, 118]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340448512Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340468Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Max found gen is 119","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340484388Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Generation requested: 120 >= found:119","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340507856Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Next flush: 209","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340524185Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340540313Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"":"downstairs","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340557362Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:20:29.340572881Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"client":"0","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340591889Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"client":"1","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340608738Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"client":"2","":"downstairs","session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340626036Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340642335Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"c87dc9d6-f2b1-4000-aae4-0cfd55f7c7f9 is now active with session: bf306c22-d4b8-4f81-a9bb-c5ea47267dab","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340660423Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340675702Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686,"session_id":"bf306c22-d4b8-4f81-a9bb-c5ea47267dab"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.34069385Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686} {"msg":"Activated sub_volume c87dc9d6-f2b1-4000-aae4-0cfd55f7c7f9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:29.340708009Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 526 len 16384 data: 7 7 8 9 02/30 Read block 451 len 12288 03/30 Read block 8 len 24576 04/30 Write block 563 len 4096 data: 5 05/30 Flush 06/30 Write block 289 len 4096 data: 6 07/30 Write block 503 len 16384 data: 7 8 8 8 08/30 Read block 412 len 24576 09/30 Write block 554 len 20480 data: 9 8 8 4 3 10/30 Write block 219 len 16384 data: 11 11 12 11 11/30 Write block 522 len 12288 data: 10 8 9 12/30 Read block 229 len 4096 13/30 Write block 443 len 8192 data: 11 11 14/30 Flush 15/30 Write block 183 len 8192 data: 11 8 16/30 Read block 32 len 28672 17/30 Write block 159 len 40960 data: 10 14 18 16 14 13 12 14 11 9 18/30 Read block 521 len 36864 19/30 Write block 587 len 32768 data: 9 7 10 12 12 11 11 11 20/30 Write block 397 len 40960 data: 11 10 10 8 9 8 9 7 9 10 21/30 Write block 271 len 4096 data: 12 22/30 Read block 275 len 32768 23/30 Flush 24/30 Write block 428 len 36864 data: 10 9 8 10 11 12 11 11 12 25/30 Read block 243 len 24576 26/30 Flush 27/30 Write block 548 len 8192 data: 8 8 28/30 Write block 486 len 28672 data: 11 11 8 7 7 10 10 29/30 Read block 314 len 40960 30/30 Write block 471 len 40960 data: 11 9 8 9 9 6 4 4 5 9 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 1 Done Sent Done false 1006 Acked Write 4 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Read 1 Done Sent Done false 1012 Acked Write 2 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 2 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Write 10 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 28 1 30 Done 29 2 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:29.359040651Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1686} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 208 208 208 F F F 1 020-039 118 118 118 208 208 208 F F F 2 040-059 118 118 118 208 208 208 F F F 3 060-079 118 118 118 208 208 208 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 113 113 113 202 202 202 F F F 6 120-139 118 118 118 208 208 208 F F F 7 140-159 120 118 120 211 208 211 F F F <--- 8 160-179 120 118 120 211 208 211 F F F <--- 9 180-199 120 118 120 211 208 211 F F F <--- 10 200-219 120 118 120 210 208 210 F F F <--- 11 220-239 120 118 120 210 208 210 F F F <--- 12 240-259 118 118 118 208 208 208 F F F 13 260-279 120 118 120 211 208 211 F F F <--- 14 280-299 120 118 120 210 208 210 F F F <--- 15 300-319 118 118 118 208 208 208 F F F 16 320-339 116 116 116 204 204 204 F F F 17 340-359 116 116 116 204 204 204 F F F 18 360-379 118 118 118 208 208 208 F F F 19 380-399 120 118 120 211 208 211 F F F <--- 20 400-419 120 117 120 211 207 211 F F F <--- 21 420-439 120 112 120 212 199 212 F F F <--- 22 440-459 120 114 120 210 203 210 F F F <--- 23 460-479 116 116 116 204 204 204 T F T <--- 24 480-499 116 116 116 204 204 204 T F T <--- 25 500-519 120 114 120 210 203 210 F F F <--- 26 520-539 120 114 120 210 203 210 F T F <--- 27 540-559 120 116 120 210 204 210 T F T <--- 28 560-579 120 118 120 209 208 209 F F F <--- 29 580-599 120 118 120 211 208 211 F F F <--- Max gen: 120, Max flush: 212 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 208 208 208 F F F 1 020-039 118 118 118 208 208 208 F F F 2 040-059 118 118 118 208 208 208 F F F 3 060-079 118 118 118 208 208 208 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 113 113 113 202 202 202 F F F 6 120-139 118 118 118 208 208 208 F F F 7 140-159 120 120 120 211 211 211 F F F 8 160-179 120 120 120 211 211 211 F F F 9 180-199 120 120 120 211 211 211 F F F 10 200-219 120 120 120 210 210 210 F F F 11 220-239 120 120 120 210 210 210 F F F 12 240-259 118 118 118 208 208 208 F F F 13 260-279 120 120 120 211 211 211 F F F 14 280-299 120 120 120 210 210 210 F F F 15 300-319 118 118 118 208 208 208 F F F 16 320-339 116 116 116 204 204 204 F F F 17 340-359 116 116 116 204 204 204 F F F 18 360-379 118 118 118 208 208 208 F F F 19 380-399 120 120 120 211 211 211 F F F 20 400-419 120 120 120 211 211 211 F F F 21 420-439 120 120 120 212 212 212 F F F 22 440-459 120 120 120 210 210 210 F F F 23 460-479 121 121 121 213 213 213 F F F 24 480-499 121 121 121 213 213 213 F F F 25 500-519 120 120 120 210 210 210 F F F 26 520-539 120 120 120 210 210 210 F F F 27 540-559 121 121 121 213 213 213 F F F 28 560-579 120 120 120 209 209 209 F F F 29 580-599 120 120 120 211 211 211 F F F Max gen: 121, Max flush: 213 Begin loop 61 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.78943838Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:33.835158796Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.835534813Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.866994432Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.867668043Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.867724048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.867743556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.867770704Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694} {"msg":"Crucible 6dc68c7b-52d4-42ef-90fd-a86f28b37782 has session id: 0362dd5d-4e80-4bf2-8f4d-be17ea208edb","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.86827679Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Upstairs opts: Upstairs UUID: 6dc68c7b-52d4-42ef-90fd-a86f28b37782, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.868360722Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Crucible stats registered with UUID: 6dc68c7b-52d4-42ef-90fd-a86f28b37782","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.868401349Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:33.868435946Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.868652207Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"io task","client":"1","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.868694733Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.868716691Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"io task","client":"2","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.868863778Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"task":"crutest"} {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.868884337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694} {"msg":"6dc68c7b-52d4-42ef-90fd-a86f28b37782 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.86896067Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.868983028Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"client":"0","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.869003746Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"client":"1","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.869029654Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"client":"2","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.869056402Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"io task","client":"2","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.869087069Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.869120006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"io task","client":"1","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"ds_connection connected from Ok(127.0.0.1:42269)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.8693049Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"ds_connection connected from Ok(127.0.0.1:50331)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.869419Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"io task","client":"2","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"ds_connection connected from Ok(127.0.0.1:48059)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.869484864Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"io task","client":"1","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.87057057Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"client":"1","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.870609837Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"client":"2","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.870631215Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"client":"0","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871156619Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871250441Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"[0]R flush_numbers[0..12]: [208, 208, 208, 208, 184, 202, 208, 211, 211, 211, 210, 210]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871342083Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"[0]R generation[0..12]: [118, 118, 118, 118, 105, 113, 118, 120, 120, 120, 120, 120]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.87136668Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871387149Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"[1]R flush_numbers[0..12]: [208, 208, 208, 208, 184, 202, 208, 211, 211, 211, 210, 210]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871406267Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"[1]R generation[0..12]: [118, 118, 118, 118, 105, 113, 118, 120, 120, 120, 120, 120]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871425295Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871445054Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"[2]R flush_numbers[0..12]: [208, 208, 208, 208, 184, 202, 208, 211, 211, 211, 210, 210]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871464032Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"[2]R generation[0..12]: [118, 118, 118, 118, 105, 113, 118, 120, 120, 120, 120, 120]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.87148278Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871502289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871518667Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871544785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Next flush: 214","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871562314Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871578532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"":"downstairs","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871595871Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871611249Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"client":"0","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871628448Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"client":"1","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871645426Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"client":"2","":"downstairs","session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871662695Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871678853Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"6dc68c7b-52d4-42ef-90fd-a86f28b37782 is now active with session: 0362dd5d-4e80-4bf2-8f4d-be17ea208edb","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871696652Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.87171205Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694,"session_id":"0362dd5d-4e80-4bf2-8f4d-be17ea208edb"} {"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871728569Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694} {"msg":"Activated sub_volume 6dc68c7b-52d4-42ef-90fd-a86f28b37782","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:33.871742708Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 545 len 20480 data: 7 7 6 9 9 02/30 Read block 525 len 20480 03/30 Read block 409 len 20480 04/30 Write block 527 len 36864 data: 8 9 10 13 12 11 11 10 9 05/30 Write block 115 len 4096 data: 6 06/30 Read block 405 len 12288 07/30 Read block 553 len 8192 08/30 Read block 46 len 8192 09/30 Read block 549 len 16384 10/30 Read block 440 len 12288 11/30 Flush 12/30 Write block 520 len 16384 data: 7 9 11 9 13/30 Write block 380 len 36864 data: 11 11 9 8 13 10 8 7 9 14/30 Read block 467 len 20480 15/30 Read block 162 len 20480 16/30 Write block 220 len 20480 data: 12 13 12 7 9 17/30 Read block 85 len 16384 18/30 Read block 103 len 28672 19/30 Read block 570 len 16384 20/30 Read block 422 len 4096 21/30 Read block 495 len 16384 22/30 Read block 317 len 28672 23/30 Read block 556 len 36864 24/30 Write block 356 len 36864 data: 12 12 12 11 12 14 14 11 12 25/30 Read block 316 len 40960 26/30 Read block 214 len 16384 27/30 Read block 531 len 24576 28/30 Write block 330 len 8192 data: 7 9 29/30 Read block 10 len 40960 30/30 Write block 308 len 4096 data: 9 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Read 5 Sent Done Done false 1003 Acked Write 9 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Read 3 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 1 30 Done 2 29 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:33.891184904Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1694} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 208 208 208 F F F 1 020-039 118 118 118 208 208 208 F F F 2 040-059 118 118 118 208 208 208 F F F 3 060-079 118 118 118 208 208 208 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 113 122 122 202 214 214 F F F <--- 6 120-139 118 118 118 208 208 208 F F F 7 140-159 120 120 120 211 211 211 F F F 8 160-179 120 120 120 211 211 211 F F F 9 180-199 120 120 120 211 211 211 F F F 10 200-219 120 120 120 210 210 210 F F F 11 220-239 120 120 120 210 210 210 F T T <--- 12 240-259 118 118 118 208 208 208 F F F 13 260-279 120 120 120 211 211 211 F F F 14 280-299 120 120 120 210 210 210 F F F 15 300-319 118 118 118 208 208 208 F T T <--- 16 320-339 116 116 116 204 204 204 F T T <--- 17 340-359 116 116 116 204 204 204 F T T <--- 18 360-379 118 118 118 208 208 208 F T T <--- 19 380-399 120 120 120 211 211 211 F T T <--- 20 400-419 120 120 120 211 211 211 F F F 21 420-439 120 120 120 212 212 212 F F F 22 440-459 120 120 120 210 210 210 F F F 23 460-479 121 121 121 213 213 213 F F F 24 480-499 121 121 121 213 213 213 F F F 25 500-519 120 120 120 210 210 210 F F F 26 520-539 120 122 122 210 214 214 F T T <--- 27 540-559 121 122 122 213 214 214 T F F <--- 28 560-579 120 120 120 209 209 209 F F F 29 580-599 120 120 120 211 211 211 F F F Max gen: 122, Max flush: 214 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 208 208 208 F F F 1 020-039 118 118 118 208 208 208 F F F 2 040-059 118 118 118 208 208 208 F F F 3 060-079 118 118 118 208 208 208 F F F 4 080-099 105 105 105 184 184 184 F F F 5 100-119 122 122 122 214 214 214 F F F 6 120-139 118 118 118 208 208 208 F F F 7 140-159 120 120 120 211 211 211 F F F 8 160-179 120 120 120 211 211 211 F F F 9 180-199 120 120 120 211 211 211 F F F 10 200-219 120 120 120 210 210 210 F F F 11 220-239 123 123 123 215 215 215 F F F 12 240-259 118 118 118 208 208 208 F F F 13 260-279 120 120 120 211 211 211 F F F 14 280-299 120 120 120 210 210 210 F F F 15 300-319 123 123 123 215 215 215 F F F 16 320-339 123 123 123 215 215 215 F F F 17 340-359 123 123 123 215 215 215 F F F 18 360-379 123 123 123 215 215 215 F F F 19 380-399 123 123 123 215 215 215 F F F 20 400-419 120 120 120 211 211 211 F F F 21 420-439 120 120 120 212 212 212 F F F 22 440-459 120 120 120 210 210 210 F F F 23 460-479 121 121 121 213 213 213 F F F 24 480-499 121 121 121 213 213 213 F F F 25 500-519 120 120 120 210 210 210 F F F 26 520-539 123 123 123 215 215 215 F F F 27 540-559 122 122 122 214 214 214 F F F 28 560-579 120 120 120 209 209 209 F F F 29 580-599 120 120 120 211 211 211 F F F Max gen: 123, Max flush: 215 Begin loop 62 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.319022167Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.363805634Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.364576337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.364624823Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.364646121Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.364668989Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702} {"msg":"Crucible 4ede6c67-19c4-41ff-9a52-984f562ccee8 has session id: 19cc6813-47a7-4344-ba67-a4be6c889f17","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365159667Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Upstairs opts: Upstairs UUID: 4ede6c67-19c4-41ff-9a52-984f562ccee8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365277906Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Crucible stats registered with UUID: 4ede6c67-19c4-41ff-9a52-984f562ccee8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365314713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:38.365340991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365499667Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"0","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365537443Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365556292Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365699859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"task":"crutest"} {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365722597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702} {"msg":"4ede6c67-19c4-41ff-9a52-984f562ccee8 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365737226Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365752815Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"client":"0","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365769713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365786632Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"client":"2","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365815999Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365849086Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"0","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365882264Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"ds_connection connected from Ok(127.0.0.1:49205)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365915261Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"0","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"ds_connection connected from Ok(127.0.0.1:43582)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.365949118Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:38.366104194Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:38.36615223Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:38.366187577Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.366221404Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.366248601Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.367330147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"client":"2","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.367375453Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"client":"0","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.367993089Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:38.368030526Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.367124864Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"ds_connection connected from Ok(127.0.0.1:33875)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.36739752Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.368941776Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"[0]R flush_numbers[0..12]: [208, 208, 208, 208, 184, 214, 208, 211, 211, 211, 210, 215]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369611637Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"[0]R generation[0..12]: [118, 118, 118, 118, 105, 122, 118, 120, 120, 120, 120, 123]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369641985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369663243Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"[1]R flush_numbers[0..12]: [208, 208, 208, 208, 184, 214, 208, 211, 211, 211, 210, 215]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369682531Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"[1]R generation[0..12]: [118, 118, 118, 118, 105, 122, 118, 120, 120, 120, 120, 123]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369701689Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369721218Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"[2]R flush_numbers[0..12]: [208, 208, 208, 208, 184, 214, 208, 211, 211, 211, 210, 215]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369740146Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"[2]R generation[0..12]: [118, 118, 118, 118, 105, 122, 118, 120, 120, 120, 120, 123]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369759164Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369778793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369795191Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.36981232Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Next flush: 216","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369833628Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369850387Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"":"downstairs","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369867515Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","Wait for a query_work_queue command to finish before sending IO pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369891563Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"client":"0","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369908991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"client":"1","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.36992584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"client":"2","":"downstairs","session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369943019Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369959157Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"4ede6c67-19c4-41ff-9a52-984f562ccee8 is now active with session: 19cc6813-47a7-4344-ba67-a4be6c889f17","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369976816Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.369992214Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702,"session_id":"19cc6813-47a7-4344-ba67-a4be6c889f17"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.370017212Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702} {"msg":"Activated sub_volume 4ede6c67-19c4-41ff-9a52-984f562ccee8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:20:48.37003584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 69 len 20480 data: 11 10 9 10 9 02/30 Write block 221 len 16384 data: 14 13 8 10 03/30 Read block 305 len 4096 04/30 Write block 417 len 20480 data: 7 8 9 11 9 05/30 Write block 61 len 16384 data: 5 7 7 10 06/30 Write block 145 len 20480 data: 9 6 6 9 10 07/30 Write block 581 len 16384 data: 6 6 6 6 08/30 Write block 313 len 20480 data: 9 9 8 7 7 09/30 Write block 504 len 36864 data: 9 9 9 7 8 10 10 11 11 10/30 Read block 139 len 24576 11/30 Write block 435 len 32768 data: 12 13 12 10 9 10 12 11 12/30 Read block 165 len 16384 13/30 Read block 489 len 16384 14/30 Read block 511 len 16384 15/30 Flush 16/30 Write block 498 len 32768 data: 6 7 7 8 6 8 10 10 17/30 Read block 315 len 32768 18/30 Write block 304 len 20480 data: 12 13 12 12 10 19/30 Read block 167 len 40960 20/30 Read block 25 len 12288 21/30 Write block 472 len 4096 data: 10 22/30 Write block 577 len 4096 data: 7 23/30 Write block 11 len 20480 data: 7 8 9 7 7 24/30 Flush 25/30 Write block 63 len 28672 data: 8 11 9 10 10 12 12 26/30 Write block 391 len 36864 data: 6 5 4 4 7 11 12 11 11 27/30 Write block 7 len 4096 data: 10 28/30 Write block 83 len 4096 data: 13 29/30 Read block 517 len 36864 30/30 Write block 440 len 40960 data: 11 13 12 12 12 11 12 14 13 16 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Write 5 Done Sent Done false 1006 Acked Write 4 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Read 6 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Read 4 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 8 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Read 10 Done Sent Done false 1019 Acked Read 3 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Write 1 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 1 28 Done 29 4 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:20:48.387775134Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1702} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 124 118 124 217 208 217 T F T <--- 1 020-039 118 118 118 208 208 208 F F F 2 040-059 118 118 118 208 208 208 F F F 3 060-079 124 118 124 216 208 216 T T T <--- 4 080-099 105 105 105 184 184 184 T F T <--- 5 100-119 122 122 122 214 214 214 F F F 6 120-139 118 118 118 208 208 208 F F F 7 140-159 124 120 124 216 211 216 F F F <--- 8 160-179 120 120 120 211 211 211 F F F 9 180-199 120 120 120 211 211 211 F F F 10 200-219 120 120 120 210 210 210 F F F 11 220-239 124 123 124 216 215 216 F T F <--- 12 240-259 118 118 118 208 208 208 F F F 13 260-279 120 120 120 211 211 211 F F F 14 280-299 120 120 120 210 210 210 F F F 15 300-319 124 123 124 217 215 217 F F F <--- 16 320-339 123 123 123 215 215 215 F F F 17 340-359 123 123 123 215 215 215 F F F 18 360-379 123 123 123 215 215 215 F F F 19 380-399 123 123 123 215 215 215 T F T <--- 20 400-419 124 120 124 216 211 216 F T F <--- 21 420-439 124 120 124 216 212 216 F T F <--- 22 440-459 124 120 124 216 210 216 T F T <--- 23 460-479 124 121 124 217 213 217 F F F <--- 24 480-499 124 121 124 217 213 217 F F F <--- 25 500-519 124 120 124 217 210 217 F F F <--- 26 520-539 123 123 123 215 215 215 F F F 27 540-559 122 122 122 214 214 214 F F F 28 560-579 124 120 124 217 209 217 F F F <--- 29 580-599 124 120 124 216 211 216 F F F <--- Max gen: 124, Max flush: 217 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 218 218 218 F F F 1 020-039 118 118 118 208 208 208 F F F 2 040-059 118 118 118 208 208 208 F F F 3 060-079 125 125 125 218 218 218 F F F 4 080-099 125 125 125 218 218 218 F F F 5 100-119 122 122 122 214 214 214 F F F 6 120-139 118 118 118 208 208 208 F F F 7 140-159 124 124 124 216 216 216 F F F 8 160-179 120 120 120 211 211 211 F F F 9 180-199 120 120 120 211 211 211 F F F 10 200-219 120 120 120 210 210 210 F F F 11 220-239 124 124 124 216 216 216 F F F 12 240-259 118 118 118 208 208 208 F F F 13 260-279 120 120 120 211 211 211 F F F 14 280-299 120 120 120 210 210 210 F F F 15 300-319 124 124 124 217 217 217 F F F 16 320-339 123 123 123 215 215 215 F F F 17 340-359 123 123 123 215 215 215 F F F 18 360-379 123 123 123 215 215 215 F F F 19 380-399 125 125 125 218 218 218 F F F 20 400-419 124 124 124 216 216 216 F F F 21 420-439 124 124 124 216 216 216 F F F 22 440-459 125 125 125 218 218 218 F F F 23 460-479 124 124 124 217 217 217 F F F 24 480-499 124 124 124 217 217 217 F F F 25 500-519 124 124 124 217 217 217 F F F 26 520-539 123 123 123 215 215 215 F F F 27 540-559 122 122 122 214 214 214 F F F 28 560-579 124 124 124 217 217 217 F F F 29 580-599 124 124 124 216 216 216 F F F Max gen: 125, Max flush: 218 Begin loop 63 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.811914628Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.856497674Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.857280495Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.85733337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.857359198Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.857387026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710} {"msg":"Crucible 95e902c5-ceb2-4e09-ac84-be8653972376 has session id: 83f51ca5-5e9c-4984-9b71-ef3f79b2d58a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.857839286Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Upstairs opts: Upstairs UUID: 95e902c5-ceb2-4e09-ac84-be8653972376, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.857897741Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Crucible stats registered with UUID: 95e902c5-ceb2-4e09-ac84-be8653972376","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.857928549Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:02.857956196Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858228203Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"io task","client":"1","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858275088Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"io task","client":"0","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858299416Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858480131Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"task":"crutest"} {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858504928Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710} {"msg":"95e902c5-ceb2-4e09-ac84-be8653972376 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858583742Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.85860391Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"client":"0","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858626418Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"client":"1","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858661325Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"client":"2","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858694702Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858728569Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"io task","client":"0","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858761806Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"io task","client":"1","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"ds_connection connected from Ok(127.0.0.1:37119)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858794883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"io task","client":"1","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"ds_connection connected from Ok(127.0.0.1:65149)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.858842919Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"io task","client":"0","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"ds_connection connected from Ok(127.0.0.1:63391)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.85905592Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.860101439Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"client":"0","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.860226689Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"client":"1","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.860251796Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"client":"2","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.860718246Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.860791939Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"[0]R flush_numbers[0..12]: [218, 208, 208, 218, 218, 214, 208, 216, 211, 211, 210, 216]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.860904059Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"[0]R generation[0..12]: [125, 118, 118, 125, 125, 122, 118, 124, 120, 120, 120, 124]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.860928767Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.860948966Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"[1]R flush_numbers[0..12]: [218, 208, 208, 218, 218, 214, 208, 216, 211, 211, 210, 216]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.860971644Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"[1]R generation[0..12]: [125, 118, 118, 125, 125, 122, 118, 124, 120, 120, 120, 124]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.860990992Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.86101076Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"[2]R flush_numbers[0..12]: [218, 208, 208, 218, 218, 214, 208, 216, 211, 211, 210, 216]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861030248Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"[2]R generation[0..12]: [125, 118, 118, 125, 125, 122, 118, 124, 120, 120, 120, 124]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861049207Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861068815Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861097613Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:21:02.861115171Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Next flush: 219","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861138829Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861154968Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"":"downstairs","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861172026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861187675Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"client":"0","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861204743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"client":"1","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861221752Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"client":"2","":"downstairs","session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.86123864Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861256889Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"95e902c5-ceb2-4e09-ac84-be8653972376 is now active with session: 83f51ca5-5e9c-4984-9b71-ef3f79b2d58a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861274967Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861290206Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710,"session_id":"83f51ca5-5e9c-4984-9b71-ef3f79b2d58a"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861306874Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710} {"msg":"Activated sub_volume 95e902c5-ceb2-4e09-ac84-be8653972376","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:02.861320713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 113 len 40960 data: 9 8 7 6 9 10 8 8 9 9 02/30 Write block 375 len 8192 data: 9 10 03/30 Read block 67 len 24576 04/30 Read block 190 len 20480 05/30 Read block 572 len 20480 06/30 Read block 448 len 32768 07/30 Read block 244 len 40960 08/30 Write block 430 len 40960 data: 9 11 12 13 12 13 14 13 11 10 09/30 Write block 347 len 28672 data: 9 10 10 10 7 10 11 10/30 Read block 179 len 12288 11/30 Read block 2 len 12288 12/30 Write block 419 len 32768 data: 10 12 10 8 7 7 7 9 13/30 Read block 49 len 32768 14/30 Read block 244 len 20480 15/30 Read block 60 len 12288 16/30 Read block 187 len 8192 17/30 Write block 544 len 24576 data: 9 8 8 7 10 10 18/30 Write block 81 len 32768 data: 10 12 14 12 10 10 7 6 19/30 Write block 149 len 24576 data: 11 12 12 10 10 8 20/30 Read block 1 len 28672 21/30 Write block 218 len 28672 data: 11 12 13 15 14 9 11 22/30 Write block 594 len 24576 data: 12 12 12 9 8 4 23/30 Read block 335 len 8192 24/30 Read block 399 len 8192 25/30 Read block 586 len 32768 26/30 Read block 319 len 4096 27/30 Write block 276 len 32768 data: 14 12 9 7 9 10 9 9 28/30 Read block 526 len 32768 29/30 Read block 145 len 12288 30/30 Write block 9 len 36864 data: 8 10 8 9 10 8 8 7 6 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Read 5 Done Done Done false 1004 Acked Read 5 Done Done Done false 1005 Acked Read 8 Done Done Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Write 10 Done Sent Done false 1008 Acked Write 7 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Read 3 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Read 8 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 24 1 26 Done 29 6 29 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:02.881502794Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1710} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 218 218 218 T F T <--- 1 020-039 118 118 118 208 208 208 F F F 2 040-059 118 118 118 208 208 208 F F F 3 060-079 125 125 125 218 218 218 F F F 4 080-099 125 125 125 218 218 218 T F T <--- 5 100-119 122 122 122 214 214 214 T T T 6 120-139 118 118 118 208 208 208 T T T 7 140-159 124 124 124 216 216 216 T F T <--- 8 160-179 120 120 120 211 211 211 F F F 9 180-199 120 120 120 211 211 211 F F F 10 200-219 120 120 120 210 210 210 T F T <--- 11 220-239 124 124 124 216 216 216 T F T <--- 12 240-259 118 118 118 208 208 208 F F F 13 260-279 120 120 120 211 211 211 T F T <--- 14 280-299 120 120 120 210 210 210 T F T <--- 15 300-319 124 124 124 217 217 217 F F F 16 320-339 123 123 123 215 215 215 F F F 17 340-359 123 123 123 215 215 215 T F T <--- 18 360-379 123 123 123 215 215 215 T T T 19 380-399 125 125 125 218 218 218 F F F 20 400-419 124 124 124 216 216 216 T F T <--- 21 420-439 124 124 124 216 216 216 T F T <--- 22 440-459 125 125 125 218 218 218 F F F 23 460-479 124 124 124 217 217 217 F F F 24 480-499 124 124 124 217 217 217 F F F 25 500-519 124 124 124 217 217 217 F F F 26 520-539 123 123 123 215 215 215 F F F 27 540-559 122 122 122 214 214 214 T F T <--- 28 560-579 124 124 124 217 217 217 F F F 29 580-599 124 124 124 216 216 216 T F T <--- Max gen: 125, Max flush: 218 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 219 219 219 F F F 1 020-039 118 118 118 208 208 208 F F F 2 040-059 118 118 118 208 208 208 F F F 3 060-079 125 125 125 218 218 218 F F F 4 080-099 126 126 126 219 219 219 F F F 5 100-119 126 126 126 219 219 219 F F F 6 120-139 126 126 126 219 219 219 F F F 7 140-159 126 126 126 219 219 219 F F F 8 160-179 120 120 120 211 211 211 F F F 9 180-199 120 120 120 211 211 211 F F F 10 200-219 126 126 126 219 219 219 F F F 11 220-239 126 126 126 219 219 219 F F F 12 240-259 118 118 118 208 208 208 F F F 13 260-279 126 126 126 219 219 219 F F F 14 280-299 126 126 126 219 219 219 F F F 15 300-319 124 124 124 217 217 217 F F F 16 320-339 123 123 123 215 215 215 F F F 17 340-359 126 126 126 219 219 219 F F F 18 360-379 126 126 126 219 219 219 F F F 19 380-399 125 125 125 218 218 218 F F F 20 400-419 126 126 126 219 219 219 F F F 21 420-439 126 126 126 219 219 219 F F F 22 440-459 125 125 125 218 218 218 F F F 23 460-479 124 124 124 217 217 217 F F F 24 480-499 124 124 124 217 217 217 F F F 25 500-519 124 124 124 217 217 217 F F F 26 520-539 123 123 123 215 215 215 F F F 27 540-559 126 126 126 219 219 219 F F F 28 560-579 124 124 124 217 217 217 F F F 29 580-599 126 126 126 219 219 219 F F F Max gen: 126, Max flush: 219 Begin loop 64 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.325064548Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.369903Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.371029002Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.371087217Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.371107985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.371135843Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718} {"msg":"Crucible 623128a1-85e7-4922-b6d7-b57230a5bef0 has session id: b2d099e0-4168-48ff-bdb5-ffd3766bca4b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.371725221Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Upstairs opts: Upstairs UUID: 623128a1-85e7-4922-b6d7-b57230a5bef0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.371767528Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Crucible stats registered with UUID: 623128a1-85e7-4922-b6d7-b57230a5bef0","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.371787716Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:07.371807834Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372050123Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372114828Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"io task","client":"0","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372156614Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"io task","client":"2","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372419841Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"task":"crutest"} {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372456508Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718} {"msg":"623128a1-85e7-4922-b6d7-b57230a5bef0 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.37253971Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372567168Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"client":"0","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372614794Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"client":"1","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372652941Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"client":"2","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372690477Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"io task","client":"0","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372730504Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"io task","client":"2","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.37277085Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"ds_connection connected from Ok(127.0.0.1:45580)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372808737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"io task","client":"0","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"ds_connection connected from Ok(127.0.0.1:38458)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.372847014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"io task","client":"2","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"ds_connection connected from Ok(127.0.0.1:49150)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.37288727Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.374004363Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"client":"0","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.374129452Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"client":"1","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.374621969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.374697272Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.375619652Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"client":"2","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"[0]R flush_numbers[0..12]: [219, 208, 208, 218, 219, 219, 219, 219, 211, 211, 219, 219]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376115199Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"[0]R generation[0..12]: [126, 118, 118, 125, 126, 126, 126, 126, 120, 120, 126, 126]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376139597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376160005Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"[1]R flush_numbers[0..12]: [219, 208, 208, 218, 219, 219, 219, 219, 211, 211, 219, 219]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376181983Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"[1]R generation[0..12]: [126, 118, 118, 125, 126, 126, 126, 126, 120, 120, 126, 126]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.37621399Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376249937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"[2]R flush_numbers[0..12]: [219, 208, 208, 218, 219, 219, 219, 219, 211, 211, 219, 219]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376284444Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"[2]R generation[0..12]: [126, 118, 118, 125, 126, 126, 126, 126, 120, 120, 126, 126]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376320251Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376347188Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Max found gen is 127","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376371806Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 128 >= found:127","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376407253Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Next flush: 220","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376438161Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376458489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"":"downstairs","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376479337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376505485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"client":"0","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376537152Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"client":"1","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.37655771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"client":"2","":"downstairs","session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376596567Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376627144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"623128a1-85e7-4922-b6d7-b57230a5bef0 is now active with session: b2d099e0-4168-48ff-bdb5-ffd3766bca4b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376655342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.37667362Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718,"session_id":"b2d099e0-4168-48ff-bdb5-ffd3766bca4b"} {"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:21:07.376701068Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718} {"msg":"Activated sub_volume 623128a1-85e7-4922-b6d7-b57230a5bef0","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:07.376726466Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 187 len 36864 data: 12 11 13 10 10 11 12 11 10 02/30 Read block 72 len 24576 03/30 Read block 532 len 32768 04/30 Read block 102 len 16384 05/30 Read block 520 len 20480 06/30 Write block 287 len 12288 data: 8 8 7 07/30 Write block 90 len 16384 data: 6 7 8 8 08/30 Write block 195 len 4096 data: 11 09/30 Read block 468 len 4096 10/30 Read block 375 len 12288 11/30 Read block 46 len 32768 12/30 Read block 305 len 12288 13/30 Read block 183 len 4096 14/30 Read block 301 len 32768 15/30 Write block 568 len 20480 data: 9 12 11 9 8 16/30 Read block 557 len 28672 17/30 Write block 564 len 28672 data: 6 6 7 8 10 13 12 18/30 Flush 19/30 Write block 96 len 24576 data: 10 12 12 10 10 11 20/30 Write block 340 len 32768 data: 11 9 8 7 7 6 7 10 21/30 Write block 499 len 8192 data: 8 8 22/30 Write block 196 len 40960 data: 14 15 17 17 15 15 12 12 10 9 23/30 Read block 465 len 28672 24/30 Read block 298 len 16384 25/30 Write block 37 len 20480 data: 7 7 8 9 7 26/30 Write block 36 len 16384 data: 7 8 8 9 27/30 Read block 144 len 16384 28/30 Write block 327 len 28672 data: 8 9 9 8 10 8 8 29/30 Write block 150 len 24576 data: 13 13 11 11 9 10 30/30 Write block 427 len 24576 data: 11 11 10 10 12 13 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Read 8 Done Done Sent false 1003 Acked Read 4 Done Done Sent false 1004 Acked Read 5 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Write 4 Done Done Sent false 1007 Acked Write 1 Done Done Sent false 1008 Acked Read 1 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Write 5 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Write 7 Done Done Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 28 32 Done 28 28 2 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) Some(JobId(1017)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:07.395920393Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1718} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 219 219 219 F F F 1 020-039 118 118 118 208 208 208 T T F <--- 2 040-059 118 118 118 208 208 208 T T F <--- 3 060-079 125 125 125 218 218 218 F F F 4 080-099 128 128 126 220 220 219 T T F <--- 5 100-119 126 126 126 219 219 219 T T F <--- 6 120-139 126 126 126 219 219 219 F F F 7 140-159 126 126 126 219 219 219 T T F <--- 8 160-179 120 120 120 211 211 211 F F F 9 180-199 128 128 120 220 220 211 T T T <--- 10 200-219 126 126 126 219 219 219 T T F <--- 11 220-239 126 126 126 219 219 219 F F F 12 240-259 118 118 118 208 208 208 F F F 13 260-279 126 126 126 219 219 219 F F F 14 280-299 128 128 126 220 220 219 F F F <--- 15 300-319 124 124 124 217 217 217 F F F 16 320-339 123 123 123 215 215 215 T T F <--- 17 340-359 126 126 126 219 219 219 T T F <--- 18 360-379 126 126 126 219 219 219 F F F 19 380-399 125 125 125 218 218 218 F F F 20 400-419 126 126 126 219 219 219 F F F 21 420-439 126 126 126 219 219 219 T T F <--- 22 440-459 125 125 125 218 218 218 F F F 23 460-479 124 124 124 217 217 217 F F F 24 480-499 124 124 124 217 217 217 T T F <--- 25 500-519 124 124 124 217 217 217 T T F <--- 26 520-539 123 123 123 215 215 215 F F F 27 540-559 126 126 126 219 219 219 F F F 28 560-579 128 128 124 220 220 217 F F F <--- 29 580-599 126 126 126 219 219 219 F F F Max gen: 128, Max flush: 220 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 219 219 219 F F F 1 020-039 129 129 129 221 221 221 F F F 2 040-059 129 129 129 221 221 221 F F F 3 060-079 125 125 125 218 218 218 F F F 4 080-099 129 129 129 221 221 221 F F F 5 100-119 129 129 129 221 221 221 F F F 6 120-139 126 126 126 219 219 219 F F F 7 140-159 129 129 129 221 221 221 F F F 8 160-179 120 120 120 211 211 211 F F F 9 180-199 129 129 129 221 221 221 F F F 10 200-219 129 129 129 221 221 221 F F F 11 220-239 126 126 126 219 219 219 F F F 12 240-259 118 118 118 208 208 208 F F F 13 260-279 126 126 126 219 219 219 F F F 14 280-299 128 128 128 220 220 220 F F F 15 300-319 124 124 124 217 217 217 F F F 16 320-339 129 129 129 221 221 221 F F F 17 340-359 129 129 129 221 221 221 F F F 18 360-379 126 126 126 219 219 219 F F F 19 380-399 125 125 125 218 218 218 F F F 20 400-419 126 126 126 219 219 219 F F F 21 420-439 129 129 129 221 221 221 F F F 22 440-459 125 125 125 218 218 218 F F F 23 460-479 124 124 124 217 217 217 F F F 24 480-499 129 129 129 221 221 221 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 123 123 123 215 215 215 F F F 27 540-559 126 126 126 219 219 219 F F F 28 560-579 128 128 128 220 220 220 F F F 29 580-599 126 126 126 219 219 219 F F F Max gen: 129, Max flush: 221 Begin loop 65 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.821119495Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.865743906Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.866492841Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.866540807Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.866557485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.866579504Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726} {"msg":"Crucible 8b11150b-8b3e-4511-84cb-407fa3e253ed has session id: 4765f0ca-68da-41d6-a6cf-9afc6c67f2ec","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867102698Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Upstairs opts: Upstairs UUID: 8b11150b-8b3e-4511-84cb-407fa3e253ed, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867161693Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Crucible stats registered with UUID: 8b11150b-8b3e-4511-84cb-407fa3e253ed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.86718999Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:11.867207709Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.86741688Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867456087Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"io task","client":"0","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867475415Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"io task","client":"2","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867717274Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"task":"crutest"} {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867750381Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726} {"msg":"8b11150b-8b3e-4511-84cb-407fa3e253ed active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867809406Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867832874Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"client":"0","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867850773Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"client":"1","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867867611Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"client":"2","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.86788759Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"io task","client":"2","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867919047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867952414Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"io task","client":"0","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:47094)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.867985281Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"io task","client":"2","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:58736)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.868017318Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:64879)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.868048825Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"io task","client":"0","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.869432345Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"client":"0","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.869467792Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"client":"2","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.869918733Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"client":"1","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870103516Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870125355Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"[0]R flush_numbers[0..12]: [219, 221, 221, 218, 221, 221, 219, 221, 211, 221, 221, 219]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870413279Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"[0]R generation[0..12]: [126, 129, 129, 125, 129, 129, 126, 129, 120, 129, 129, 126]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870441267Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870461745Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"[1]R flush_numbers[0..12]: [219, 221, 221, 218, 221, 221, 219, 221, 211, 221, 221, 219]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870481263Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"[1]R generation[0..12]: [126, 129, 129, 125, 129, 129, 126, 129, 120, 129, 129, 126]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870500122Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.87051977Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"[2]R flush_numbers[0..12]: [219, 221, 221, 218, 221, 221, 219, 221, 211, 221, 221, 219]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870538808Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"[2]R generation[0..12]: [126, 129, 129, 125, 129, 129, 126, 129, 120, 129, 129, 126]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870557617Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870577005Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870593204Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870610542Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairsWait for a query_work_queue command to finish before sending IO ","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Next flush: 222","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.87063325Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870649509Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"":"downstairs","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870666557Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870681876Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"client":"0","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870700734Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"client":"1","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870717773Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"client":"2","":"downstairs","session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870734911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.87075106Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"8b11150b-8b3e-4511-84cb-407fa3e253ed is now active with session: 4765f0ca-68da-41d6-a6cf-9afc6c67f2ec","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870772248Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870788157Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726,"session_id":"4765f0ca-68da-41d6-a6cf-9afc6c67f2ec"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870804335Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726} {"msg":"Activated sub_volume 8b11150b-8b3e-4511-84cb-407fa3e253ed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:11.870818444Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 321 len 32768 data: 8 8 7 7 6 8 9 10 02/30 Write block 415 len 4096 data: 11 03/30 Flush 04/30 Write block 192 len 24576 data: 12 13 12 12 15 16 05/30 Flush 06/30 Write block 481 len 12288 data: 9 12 13 07/30 Write block 380 len 20480 data: 12 12 10 9 14 08/30 Read block 312 len 20480 09/30 Read block 133 len 8192 10/30 Write block 93 len 24576 data: 9 10 12 11 13 13 11/30 Write block 318 len 36864 data: 5 6 7 9 9 8 8 7 9 12/30 Read block 302 len 32768 13/30 Read block 380 len 36864 14/30 Write block 217 len 32768 data: 10 12 13 14 16 15 10 12 15/30 Read block 289 len 12288 16/30 Write block 550 len 32768 data: 11 10 9 10 10 9 9 5 17/30 Read block 396 len 12288 18/30 Write block 301 len 24576 data: 11 13 13 13 14 13 19/30 Write block 31 len 12288 data: 9 7 10 20/30 Read block 188 len 20480 21/30 Read block 95 len 16384 22/30 Read block 238 len 8192 23/30 Read block 128 len 36864 24/30 Write block 369 len 40960 data: 11 6 6 7 8 8 10 11 10 10 25/30 Write block 121 len 16384 data: 10 10 10 8 26/30 Write block 1 len 32768 data: 3 4 5 6 10 12 11 7 27/30 Read block 159 len 16384 28/30 Write block 232 len 16384 data: 12 9 7 6 29/30 Write block 33 len 24576 data: 11 7 7 8 9 9 30/30 Read block 115 len 28672 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Sent Done false 1001 Acked Write 1 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 6 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 3 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Write 6 Done Sent Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Read 8 Done Sent Done false 1012 Acked Read 9 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Write 8 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 4 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Write 6 Done Sent Done false 1029 Acked Read 7 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) None Some(JobId(1004)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:11.890289867Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1726} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 126 126 126 219 219 219 T F T <--- 1 020-039 129 129 129 221 221 221 T F T <--- 2 040-059 129 129 129 221 221 221 F F F 3 060-079 125 125 125 218 218 218 F F F 4 080-099 129 129 129 221 221 221 T F T <--- 5 100-119 129 129 129 221 221 221 F F F 6 120-139 126 126 126 219 219 219 T F T <--- 7 140-159 129 129 129 221 221 221 F F F 8 160-179 120 120 120 211 211 211 F F F 9 180-199 130 129 130 223 221 223 F F F <--- 10 200-219 129 129 129 221 221 221 T F T <--- 11 220-239 126 126 126 219 219 219 T F T <--- 12 240-259 118 118 118 208 208 208 F F F 13 260-279 126 126 126 219 219 219 F F F 14 280-299 128 128 128 220 220 220 F F F 15 300-319 124 124 124 217 217 217 T F T <--- 16 320-339 130 129 130 222 221 222 T F T <--- 17 340-359 129 129 129 221 221 221 F F F 18 360-379 126 126 126 219 219 219 T F T <--- 19 380-399 125 125 125 218 218 218 T F T <--- 20 400-419 130 126 130 222 219 222 F F F <--- 21 420-439 129 129 129 221 221 221 F F F 22 440-459 125 125 125 218 218 218 F F F 23 460-479 124 124 124 217 217 217 F F F 24 480-499 129 129 129 221 221 221 T F T <--- 25 500-519 129 129 129 221 221 221 F F F 26 520-539 123 123 123 215 215 215 F F F 27 540-559 126 126 126 219 219 219 T F T <--- 28 560-579 128 128 128 220 220 220 F F F 29 580-599 126 126 126 219 219 219 F F F Max gen: 130, Max flush: 223 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 131 131 131 224 224 224 F F F 1 020-039 131 131 131 224 224 224 F F F 2 040-059 129 129 129 221 221 221 F F F 3 060-079 125 125 125 218 218 218 F F F 4 080-099 131 131 131 224 224 224 F F F 5 100-119 129 129 129 221 221 221 F F F 6 120-139 131 131 131 224 224 224 F F F 7 140-159 129 129 129 221 221 221 F F F 8 160-179 120 120 120 211 211 211 F F F 9 180-199 130 130 130 223 223 223 F F F 10 200-219 131 131 131 224 224 224 F F F 11 220-239 131 131 131 224 224 224 F F F 12 240-259 118 118 118 208 208 208 F F F 13 260-279 126 126 126 219 219 219 F F F 14 280-299 128 128 128 220 220 220 F F F 15 300-319 131 131 131 224 224 224 F F F 16 320-339 131 131 131 224 224 224 F F F 17 340-359 129 129 129 221 221 221 F F F 18 360-379 131 131 131 224 224 224 F F F 19 380-399 131 131 131 224 224 224 F F F 20 400-419 130 130 130 222 222 222 F F F 21 420-439 129 129 129 221 221 221 F F F 22 440-459 125 125 125 218 218 218 F F F 23 460-479 124 124 124 217 217 217 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 123 123 123 215 215 215 F F F 27 540-559 131 131 131 224 224 224 F F F 28 560-579 128 128 128 220 220 220 F F F 29 580-599 126 126 126 219 219 219 F F F Max gen: 131, Max flush: 224 Begin loop 66 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.308344054Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.353260619Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.353992716Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.354061619Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.354084058Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.354114205Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734} {"msg":"Crucible da8c5a8a-1805-45ed-8715-49b8af7ae6ff has session id: d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.354564536Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Upstairs opts: Upstairs UUID: da8c5a8a-1805-45ed-8715-49b8af7ae6ff, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.354690595Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Crucible stats registered with UUID: da8c5a8a-1805-45ed-8715-49b8af7ae6ff","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.354733091Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:16.354762258Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.354976Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355018866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"io task","client":"0","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355044304Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"io task","client":"2","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.35520236Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"task":"crutest"} {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355231718Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734} {"msg":"da8c5a8a-1805-45ed-8715-49b8af7ae6ff active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355304591Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355332399Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"client":"0","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355365116Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"client":"1","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355396793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"client":"2","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.35543056Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"io task","client":"2","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355479616Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"io task","client":"0","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355512203Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"ds_connection connected from Ok(127.0.0.1:63855)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.35554403Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"io task","client":"2","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"ds_connection connected from Ok(127.0.0.1:58972)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355598256Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"io task","client":"0","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"ds_connection connected from Ok(127.0.0.1:34126)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.355713096Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.356980465Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"client":"2","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357016212Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"client":"0","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357089386Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"client":"1","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357555375Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357631879Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"[0]R flush_numbers[0..12]: [224, 224, 221, 218, 224, 221, 224, 221, 211, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357738889Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"[0]R generation[0..12]: [131, 131, 129, 125, 131, 129, 131, 129, 120, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357764517Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357785345Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"[1]R flush_numbers[0..12]: [224, 224, 221, 218, 224, 221, 224, 221, 211, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357804823Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"[1]R generation[0..12]: [131, 131, 129, 125, 131, 129, 131, 129, 120, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357824182Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.35784403Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"[2]R flush_numbers[0..12]: [224, 224, 221, 218, 224, 221, 224, 221, 211, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357863538Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"[2]R generation[0..12]: [131, 131, 129, 125, 131, 129, 131, 129, 120, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357886316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357906124Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357922513Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357939812Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Next flush: 225","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.35795613Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357972299Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"":"downstairs","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"Wait for a query_work_queue command to finish before sending IO } {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.357997407Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.358012785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"client":"0","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.358029924Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"client":"1","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.358046622Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"client":"2","":"downstairs","session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.358063941Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.358080489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"da8c5a8a-1805-45ed-8715-49b8af7ae6ff is now active with session: d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.358098608Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.358114137Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734,"session_id":"d6a81bbb-d62a-4ee4-9e2a-511eda8d3d03"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.358130275Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734} {"msg":"Activated sub_volume da8c5a8a-1805-45ed-8715-49b8af7ae6ff","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:16.358144394Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 370 len 8192 data: 7 7 02/30 Write block 322 len 40960 data: 10 9 9 8 10 10 11 10 9 11 03/30 Write block 256 len 16384 data: 11 13 10 10 04/30 Read block 312 len 32768 05/30 Read block 170 len 12288 06/30 Read block 334 len 4096 07/30 Read block 290 len 12288 08/30 Write block 534 len 28672 data: 11 10 13 12 11 9 11 09/30 Read block 17 len 8192 10/30 Read block 180 len 8192 11/30 Write block 7 len 4096 data: 12 12/30 Read block 469 len 28672 13/30 Flush 14/30 Write block 91 len 36864 data: 8 9 10 11 13 12 14 14 11 15/30 Write block 562 len 12288 data: 4 6 7 16/30 Read block 320 len 24576 17/30 Write block 310 len 8192 data: 12 11 18/30 Flush 19/30 Write block 113 len 40960 data: 10 9 8 7 10 11 9 9 11 11 20/30 Read block 445 len 40960 21/30 Read block 127 len 28672 22/30 Write block 315 len 12288 data: 9 8 8 23/30 Flush 24/30 Write block 359 len 8192 data: 12 13 25/30 Write block 392 len 4096 data: 6 26/30 Write block 406 len 24576 data: 11 11 11 10 10 11 27/30 Write block 159 len 4096 data: 11 28/30 Write block 172 len 32768 data: 9 9 9 8 11 10 9 8 29/30 Read block 103 len 32768 30/30 Write block 316 len 40960 data: 9 9 6 7 8 10 11 10 10 9 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Read 1 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Read 7 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 2 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Write 8 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 1 28 Done 29 4 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:16.375621881Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1734} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 131 132 225 224 225 F F F <--- 1 020-039 131 131 131 224 224 224 F F F 2 040-059 129 129 129 221 221 221 F F F 3 060-079 125 125 125 218 218 218 F F F 4 080-099 132 131 132 226 224 226 F F F <--- 5 100-119 132 129 132 227 221 227 F F F <--- 6 120-139 132 131 132 227 224 227 F F F <--- 7 140-159 129 129 129 221 221 221 T F T <--- 8 160-179 120 120 120 211 211 211 T F T <--- 9 180-199 130 130 130 223 223 223 F F F 10 200-219 131 131 131 224 224 224 F F F 11 220-239 131 131 131 224 224 224 F F F 12 240-259 132 118 132 225 208 225 F T F <--- 13 260-279 126 126 126 219 219 219 F F F 14 280-299 128 128 128 220 220 220 F F F 15 300-319 132 131 132 227 224 227 T F T <--- 16 320-339 132 131 132 225 224 225 T T T <--- 17 340-359 129 129 129 221 221 221 T F T <--- 18 360-379 132 131 132 225 224 225 T T T <--- 19 380-399 131 131 131 224 224 224 T F T <--- 20 400-419 130 130 130 222 222 222 T F T <--- 21 420-439 129 129 129 221 221 221 F F F 22 440-459 125 125 125 218 218 218 F F F 23 460-479 124 124 124 217 217 217 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 132 123 132 225 215 225 F F F <--- 27 540-559 132 131 132 225 224 225 F F F <--- 28 560-579 132 128 132 226 220 226 F F F <--- 29 580-599 126 126 126 219 219 219 F F F Max gen: 132, Max flush: 227 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 225 225 225 F F F 1 020-039 131 131 131 224 224 224 F F F 2 040-059 129 129 129 221 221 221 F F F 3 060-079 125 125 125 218 218 218 F F F 4 080-099 132 132 132 226 226 226 F F F 5 100-119 132 132 132 227 227 227 F F F 6 120-139 132 132 132 227 227 227 F F F 7 140-159 133 133 133 228 228 228 F F F 8 160-179 133 133 133 228 228 228 F F F 9 180-199 130 130 130 223 223 223 F F F 10 200-219 131 131 131 224 224 224 F F F 11 220-239 131 131 131 224 224 224 F F F 12 240-259 132 132 132 225 225 225 F F F 13 260-279 126 126 126 219 219 219 F F F 14 280-299 128 128 128 220 220 220 F F F 15 300-319 133 133 133 228 228 228 F F F 16 320-339 133 133 133 228 228 228 F F F 17 340-359 133 133 133 228 228 228 F F F 18 360-379 133 133 133 228 228 228 F F F 19 380-399 133 133 133 228 228 228 F F F 20 400-419 133 133 133 228 228 228 F F F 21 420-439 129 129 129 221 221 221 F F F 22 440-459 125 125 125 218 218 218 F F F 23 460-479 124 124 124 217 217 217 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 132 132 132 225 225 225 F F F 27 540-559 132 132 132 225 225 225 F F F 28 560-579 132 132 132 226 226 226 F F F 29 580-599 126 126 126 219 219 219 F F F Max gen: 133, Max flush: 228 Begin loop 67 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.812304943Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.856911596Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.857712136Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.857757922Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.857775501Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.857797459Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742} {"msg":"Crucible c8c1a63c-0fbc-4dce-a8d2-accd4604ee35 has session id: bfd4eaf3-38d7-4a75-a753-cf980d39c8bd","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.858291516Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Upstairs opts: Upstairs UUID: c8c1a63c-0fbc-4dce-a8d2-accd4604ee35, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.858332072Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Crucible stats registered with UUID: c8c1a63c-0fbc-4dce-a8d2-accd4604ee35","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.85835094Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:20.858367159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.858599239Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.858645645Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"io task","client":"0","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.858669583Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"io task","client":"2","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.858854467Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"task":"crutest"} {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.858886934Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742} {"msg":"c8c1a63c-0fbc-4dce-a8d2-accd4604ee35 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.858940969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.858971126Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"client":"0","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.858994544Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"client":"1","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.859036661Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"client":"2","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.859073318Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"io task","client":"2","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.859104345Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"io task","client":"0","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.859134362Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:34084)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.859166669Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:43546)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.859198987Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"io task","client":"2","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:52143)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.859410888Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"io task","client":"0","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.860384263Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"client":"0","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.86053904Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"client":"2","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.860813466Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"client":"1","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.860989151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861048076Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"[0]R flush_numbers[0..12]: [225, 224, 221, 218, 226, 227, 227, 228, 228, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861303303Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"[0]R generation[0..12]: [132, 131, 129, 125, 132, 132, 132, 133, 133, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861332471Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861353469Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"[1]R flush_numbers[0..12]: [225, 224, 221, 218, 226, 227, 227, 228, 228, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861372587Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"[1]R generation[0..12]: [132, 131, 129, 125, 132, 132, 132, 133, 133, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861391726Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861411404Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"[2]R flush_numbers[0..12]: [225, 224, 221, 218, 226, 227, 227, 228, 228, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861430592Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"[2]R generation[0..12]: [132, 131, 129, 125, 132, 132, 132, 133, 133, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.86144947Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861469259Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861485567Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861502776Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Next flush: 229","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861519164Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861535283Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"":"downstairs","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861556821Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.86157265Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"client":"0","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2025-12-24T00:21:20.861595678Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"client":"1","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861612896Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"client":"2","":"downstairs","session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861629875Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861646004Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"c8c1a63c-0fbc-4dce-a8d2-accd4604ee35 is now active with session: bfd4eaf3-38d7-4a75-a753-cf980d39c8bd","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861663752Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861679201Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742,"session_id":"bfd4eaf3-38d7-4a75-a753-cf980d39c8bd"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861695189Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742} {"msg":"Activated sub_volume c8c1a63c-0fbc-4dce-a8d2-accd4604ee35","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:20.861709058Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 268 len 8192 data: 9 11 03/30 Read block 563 len 20480 04/30 Read block 97 len 24576 05/30 Write block 425 len 32768 data: 8 10 12 12 11 11 13 14 06/30 Write block 383 len 8192 data: 10 15 07/30 Write block 281 len 36864 data: 11 10 10 9 8 9 9 9 8 08/30 Read block 228 len 8192 09/30 Write block 401 len 16384 data: 10 9 10 8 10/30 Write block 61 len 28672 data: 6 8 9 12 10 11 11 11/30 Read block 484 len 36864 12/30 Write block 401 len 36864 data: 11 10 11 9 10 12 12 12 11 13/30 Write block 336 len 8192 data: 8 9 14/30 Read block 268 len 4096 15/30 Read block 368 len 36864 16/30 Read block 111 len 40960 17/30 Read block 226 len 12288 18/30 Read block 468 len 28672 19/30 Write block 453 len 32768 data: 11 12 11 10 9 9 10 8 20/30 Read block 34 len 24576 21/30 Write block 35 len 8192 data: 8 9 22/30 Write block 594 len 24576 data: 13 13 13 10 9 5 23/30 Write block 17 len 40960 data: 7 8 10 7 9 10 11 12 12 12 24/30 Read block 484 len 32768 25/30 Read block 507 len 8192 26/30 Write block 575 len 20480 data: 8 8 8 9 8 27/30 Write block 439 len 28672 data: 11 12 14 13 13 13 12 28/30 Write block 254 len 28672 data: 8 10 12 14 11 11 11 29/30 Read block 85 len 28672 30/30 Write block 118 len 20480 data: 12 10 10 12 12 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 2 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Write 9 Done Sent Done false 1007 Acked Read 2 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Write 7 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Write 2 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Write 10 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Write 5 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 1 28 Done 28 3 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:20.880931733Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1742} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 132 132 132 225 225 225 T F T <--- 1 020-039 131 131 131 224 224 224 T F T <--- 2 040-059 129 129 129 221 221 221 F F F 3 060-079 125 125 125 218 218 218 T F T <--- 4 080-099 132 132 132 226 226 226 F F F 5 100-119 132 132 132 227 227 227 T F T <--- 6 120-139 132 132 132 227 227 227 T F T <--- 7 140-159 133 133 133 228 228 228 F F F 8 160-179 133 133 133 228 228 228 F F F 9 180-199 130 130 130 223 223 223 F F F 10 200-219 131 131 131 224 224 224 F F F 11 220-239 131 131 131 224 224 224 F F F 12 240-259 132 132 132 225 225 225 T F T <--- 13 260-279 126 126 126 219 219 219 T T T 14 280-299 128 128 128 220 220 220 T F T <--- 15 300-319 133 133 133 228 228 228 F F F 16 320-339 133 133 133 228 228 228 T F T <--- 17 340-359 133 133 133 228 228 228 F F F 18 360-379 133 133 133 228 228 228 F F F 19 380-399 133 133 133 228 228 228 T F T <--- 20 400-419 133 133 133 228 228 228 T F T <--- 21 420-439 129 129 129 221 221 221 T F T <--- 22 440-459 125 125 125 218 218 218 T F T <--- 23 460-479 124 124 124 217 217 217 T F T <--- 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 132 132 132 225 225 225 F F F 27 540-559 132 132 132 225 225 225 F F F 28 560-579 132 132 132 226 226 226 T F T <--- 29 580-599 126 126 126 219 219 219 T F T <--- Max gen: 133, Max flush: 228 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 229 229 229 F F F 1 020-039 134 134 134 229 229 229 F F F 2 040-059 129 129 129 221 221 221 F F F 3 060-079 134 134 134 229 229 229 F F F 4 080-099 132 132 132 226 226 226 F F F 5 100-119 134 134 134 229 229 229 F F F 6 120-139 134 134 134 229 229 229 F F F 7 140-159 133 133 133 228 228 228 F F F 8 160-179 133 133 133 228 228 228 F F F 9 180-199 130 130 130 223 223 223 F F F 10 200-219 131 131 131 224 224 224 F F F 11 220-239 131 131 131 224 224 224 F F F 12 240-259 134 134 134 229 229 229 F F F 13 260-279 134 134 134 229 229 229 F F F 14 280-299 134 134 134 229 229 229 F F F 15 300-319 133 133 133 228 228 228 F F F 16 320-339 134 134 134 229 229 229 F F F 17 340-359 133 133 133 228 228 228 F F F 18 360-379 133 133 133 228 228 228 F F F 19 380-399 134 134 134 229 229 229 F F F 20 400-419 134 134 134 229 229 229 F F F 21 420-439 134 134 134 229 229 229 F F F 22 440-459 134 134 134 229 229 229 F F F 23 460-479 134 134 134 229 229 229 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 132 132 132 225 225 225 F F F 27 540-559 132 132 132 225 225 225 F F F 28 560-579 134 134 134 229 229 229 F F F 29 580-599 134 134 134 229 229 229 F F F Max gen: 134, Max flush: 229 Begin loop 68 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.31480756Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.35956862Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.360329623Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.360396108Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.360416936Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.360444113Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750} {"msg":"Crucible 6afeb94a-83f4-4661-bc32-c6b99e1068d1 has session id: e04614b4-c549-48c8-98dc-70c8043f68cc","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.360893234Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Upstairs opts: Upstairs UUID: 6afeb94a-83f4-4661-bc32-c6b99e1068d1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.360952269Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Crucible stats registered with UUID: 6afeb94a-83f4-4661-bc32-c6b99e1068d1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.360979317Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:25.360996365Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361181719Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361229745Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"0","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361254353Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361369983Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"task":"crutest"} {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361394781Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750} {"msg":"6afeb94a-83f4-4661-bc32-c6b99e1068d1 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361429958Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361449826Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"client":"0","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361470854Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361505631Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"client":"2","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361545048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361586814Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"0","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361622231Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"ds_connection connected from Ok(127.0.0.1:51326)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361653048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"0","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:25.361826963Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"ds_connection connected from Ok(127.0.0.1:56099)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.361867889Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:25.361902116Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:25.361934083Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.362046844Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.36209216Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.362903589Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"client":"2","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.36300093Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"client":"0","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.363532164Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:25.363601448Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.36364168Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"ds_connection connected from Ok(127.0.0.1:61971)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.363782548Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.365421876Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"[0]R flush_numbers[0..12]: [229, 229, 221, 229, 226, 229, 229, 228, 228, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366069159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"[0]R generation[0..12]: [134, 134, 129, 134, 132, 134, 134, 133, 133, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366107326Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366133554Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"[1]R flush_numbers[0..12]: [229, 229, 221, 229, 226, 229, 229, 228, 228, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366157461Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"[1]R generation[0..12]: [134, 134, 129, 134, 132, 134, 134, 133, 133, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366181289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366205737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"[2]R flush_numbers[0..12]: [229, 229, 221, 229, 226, 229, 229, 228, 228, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366229215Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"[2]R generation[0..12]: [134, 134, 129, 134, 132, 134, 134, 133, 133, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366252733Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366276941Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Max found gen is 135","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366297299Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Generation requested: 136 >= found:135","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366318717Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Next flush: 230","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:21:35.366339045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366362994Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"":"downstairs","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366388501Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366407959Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"client":"0","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366429518Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"client":"1","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366450756Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"client":"2","":"downstairs","session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366472294Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366492362Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"6afeb94a-83f4-4661-bc32-c6b99e1068d1 is now active with session: e04614b4-c549-48c8-98dc-70c8043f68cc","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.36651439Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366533388Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750,"session_id":"e04614b4-c549-48c8-98dc-70c8043f68cc"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366553667Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750} {"msg":"Activated sub_volume 6afeb94a-83f4-4661-bc32-c6b99e1068d1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:35.366571465Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 15 len 16384 data: 9 8 8 9 02/30 Read block 535 len 24576 03/30 Read block 177 len 12288 04/30 Write block 305 len 20480 data: 15 14 13 11 9 05/30 Flush 06/30 Flush 07/30 Flush 08/30 Write block 430 len 40960 data: 12 14 15 14 13 14 15 14 12 12 09/30 Read block 377 len 24576 10/30 Write block 526 len 24576 data: 8 9 10 11 14 13 11/30 Read block 186 len 20480 12/30 Read block 440 len 20480 13/30 Read block 171 len 4096 14/30 Read block 304 len 32768 15/30 Flush 16/30 Write block 92 len 32768 data: 10 11 12 14 13 15 15 12 17/30 Read block 213 len 4096 18/30 Read block 381 len 8192 19/30 Read block 581 len 20480 20/30 Write block 424 len 16384 data: 8 9 11 13 21/30 Read block 393 len 8192 22/30 Read block 127 len 40960 23/30 Read block 170 len 24576 24/30 Read block 106 len 20480 25/30 Write block 67 len 36864 data: 12 13 13 11 10 11 10 10 12 26/30 Read block 183 len 36864 27/30 Read block 180 len 40960 28/30 Read block 577 len 20480 29/30 Write block 280 len 16384 data: 10 12 11 11 30/30 Write block 525 len 24576 data: 8 9 10 11 12 15 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 5 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 10 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Write 6 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 8 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 27 2 31 Done 28 3 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) None Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:35.387499112Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1750} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 134 136 230 229 230 F T F <--- 1 020-039 134 134 134 229 229 229 F F F 2 040-059 129 129 129 221 221 221 F F F 3 060-079 134 134 134 229 229 229 T F T <--- 4 080-099 132 132 132 226 226 226 T F T <--- 5 100-119 134 134 134 229 229 229 F F F 6 120-139 134 134 134 229 229 229 F F F 7 140-159 133 133 133 228 228 228 F F F 8 160-179 133 133 133 228 228 228 F F F 9 180-199 130 130 130 223 223 223 F F F 10 200-219 131 131 131 224 224 224 F F F 11 220-239 131 131 131 224 224 224 F F F 12 240-259 134 134 134 229 229 229 F F F 13 260-279 134 134 134 229 229 229 F F F 14 280-299 134 134 134 229 229 229 T F T <--- 15 300-319 136 133 136 230 228 230 F F F <--- 16 320-339 134 134 134 229 229 229 F F F 17 340-359 133 133 133 228 228 228 F F F 18 360-379 133 133 133 228 228 228 F F F 19 380-399 134 134 134 229 229 229 F F F 20 400-419 134 134 134 229 229 229 F F F 21 420-439 136 134 136 233 229 233 T F T <--- 22 440-459 134 134 134 229 229 229 F F F 23 460-479 134 134 134 229 229 229 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 136 132 136 233 225 233 T F T <--- 27 540-559 132 132 132 225 225 225 F F F 28 560-579 134 134 134 229 229 229 F F F 29 580-599 134 134 134 229 229 229 F F F Max gen: 136, Max flush: 233 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 230 230 230 F F F 1 020-039 134 134 134 229 229 229 F F F 2 040-059 129 129 129 221 221 221 F F F 3 060-079 137 137 137 234 234 234 F F F 4 080-099 137 137 137 234 234 234 F F F 5 100-119 134 134 134 229 229 229 F F F 6 120-139 134 134 134 229 229 229 F F F 7 140-159 133 133 133 228 228 228 F F F 8 160-179 133 133 133 228 228 228 F F F 9 180-199 130 130 130 223 223 223 F F F 10 200-219 131 131 131 224 224 224 F F F 11 220-239 131 131 131 224 224 224 F F F 12 240-259 134 134 134 229 229 229 F F F 13 260-279 134 134 134 229 229 229 F F F 14 280-299 137 137 137 234 234 234 F F F 15 300-319 136 136 136 230 230 230 F F F 16 320-339 134 134 134 229 229 229 F F F 17 340-359 133 133 133 228 228 228 F F F 18 360-379 133 133 133 228 228 228 F F F 19 380-399 134 134 134 229 229 229 F F F 20 400-419 134 134 134 229 229 229 F F F 21 420-439 137 137 137 234 234 234 F F F 22 440-459 134 134 134 229 229 229 F F F 23 460-479 134 134 134 229 229 229 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 137 137 137 234 234 234 F F F 27 540-559 132 132 132 225 225 225 F F F 28 560-579 134 134 134 229 229 229 F F F 29 580-599 134 134 134 229 229 229 F F F Max gen: 137, Max flush: 234 Begin loop 69 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.773424359Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.818864239Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.819610144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.81965883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.819675759Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.819698006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758} {"msg":"Crucible fd13c152-274b-47aa-b607-5e5ca804d0ad has session id: caac99c4-e08a-40e0-8596-3feb5125a7e5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820257938Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Upstairs opts: Upstairs UUID: fd13c152-274b-47aa-b607-5e5ca804d0ad, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820318343Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Crucible stats registered with UUID: fd13c152-274b-47aa-b607-5e5ca804d0ad","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.82034953Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:39.820375018Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820550312Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820592079Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"io task","client":"0","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820617956Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"io task","client":"1","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820798261Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"task":"crutest"} {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820836077Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758} {"msg":"fd13c152-274b-47aa-b607-5e5ca804d0ad active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820874484Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820891702Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"client":"0","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820909111Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"client":"1","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.82092593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"client":"2","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820956077Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.820976305Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"io task","client":"0","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.821007223Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"io task","client":"1","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37753)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.82115545Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:62461)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.821201726Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"io task","client":"0","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:46648)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.821237323Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"io task","client":"1","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.822585945Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"client":"0","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.822620632Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"client":"2","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.82264669Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"client":"1","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823170424Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823259126Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"[0]R flush_numbers[0..12]: [230, 229, 221, 234, 234, 229, 229, 228, 228, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.82333534Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"[0]R generation[0..12]: [136, 134, 129, 137, 137, 134, 134, 133, 133, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823360537Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823381096Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"[1]R flush_numbers[0..12]: [230, 229, 221, 234, 234, 229, 229, 228, 228, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823400184Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"[1]R generation[0..12]: [136, 134, 129, 137, 137, 134, 134, 133, 133, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823419292Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823438691Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"[2]R flush_numbers[0..12]: [230, 229, 221, 234, 234, 229, 229, 228, 228, 223, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823462178Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"[2]R generation[0..12]: [136, 134, 129, 137, 137, 134, 134, 133, 133, 130, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823481597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823501255Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823517944Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-12-24T00:21:39.823535022Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Next flush: 235","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.82355829Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823574699Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"":"downstairs","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823591717Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823607116Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"client":"0","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823624195Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"client":"1","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823641153Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"client":"2","":"downstairs","session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823658401Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.82367456Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"fd13c152-274b-47aa-b607-5e5ca804d0ad is now active with session: caac99c4-e08a-40e0-8596-3feb5125a7e5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823692389Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823707657Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758,"session_id":"caac99c4-e08a-40e0-8596-3feb5125a7e5"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823725586Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758} {"msg":"Activated sub_volume fd13c152-274b-47aa-b607-5e5ca804d0ad","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:39.823740014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 332 len 12288 data: 9 9 8 03/30 Read block 4 len 28672 04/30 Read block 311 len 36864 05/30 Flush 06/30 Write block 250 len 12288 data: 12 11 11 07/30 Read block 207 len 40960 08/30 Read block 8 len 24576 09/30 Read block 522 len 8192 10/30 Write block 346 len 4096 data: 8 11/30 Write block 190 len 4096 data: 11 12/30 Read block 154 len 16384 13/30 Flush 14/30 Write block 265 len 20480 data: 10 10 11 10 12 15/30 Write block 300 len 36864 data: 12 12 14 14 14 16 15 14 12 16/30 Write block 390 len 4096 data: 7 17/30 Write block 363 len 20480 data: 12 13 11 11 10 18/30 Read block 65 len 16384 19/30 Read block 389 len 4096 20/30 Write block 108 len 40960 data: 9 9 9 9 8 11 10 9 8 11 21/30 Write block 89 len 12288 data: 5 7 9 22/30 Write block 141 len 12288 data: 12 12 9 23/30 Read block 307 len 24576 24/30 Read block 161 len 20480 25/30 Read block 66 len 20480 26/30 Read block 40 len 20480 27/30 Write block 560 len 4096 data: 6 28/30 Read block 226 len 20480 29/30 Write block 562 len 20480 data: 5 7 8 7 8 30/30 Write block 30 len 8192 data: 10 10 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Done Done Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 3 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Write 10 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 28 2 32 Done 27 1 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) Some(JobId(1000)) Some(JobId(1012)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:39.841059435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1758} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 230 230 230 F F F 1 020-039 134 134 134 229 229 229 T F T <--- 2 040-059 129 129 129 221 221 221 F F F 3 060-079 137 137 137 234 234 234 F F F 4 080-099 137 137 137 234 234 234 T F T <--- 5 100-119 134 134 134 229 229 229 T F T <--- 6 120-139 134 134 134 229 229 229 F F F 7 140-159 133 133 133 228 228 228 T F T <--- 8 160-179 133 133 133 228 228 228 F F F 9 180-199 138 130 138 237 223 237 F F F <--- 10 200-219 131 131 131 224 224 224 F F F 11 220-239 131 131 131 224 224 224 F F F 12 240-259 138 134 138 237 229 237 F F F <--- 13 260-279 134 134 134 229 229 229 T F T <--- 14 280-299 137 137 137 234 234 234 F F F 15 300-319 136 136 136 230 230 230 T F T <--- 16 320-339 138 134 138 236 229 236 F T F <--- 17 340-359 138 133 138 237 228 237 F F F <--- 18 360-379 133 133 133 228 228 228 T F T <--- 19 380-399 134 134 134 229 229 229 T F T <--- 20 400-419 134 134 134 229 229 229 F F F 21 420-439 137 137 137 234 234 234 F F F 22 440-459 134 134 134 229 229 229 F F F 23 460-479 134 134 134 229 229 229 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 137 137 137 234 234 234 F F F 27 540-559 132 132 132 225 225 225 F F F 28 560-579 134 134 134 229 229 229 T F T <--- 29 580-599 134 134 134 229 229 229 F F F Max gen: 138, Max flush: 237 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 230 230 230 F F F 1 020-039 139 139 139 238 238 238 F F F 2 040-059 129 129 129 221 221 221 F F F 3 060-079 137 137 137 234 234 234 F F F 4 080-099 139 139 139 238 238 238 F F F 5 100-119 139 139 139 238 238 238 F F F 6 120-139 134 134 134 229 229 229 F F F 7 140-159 139 139 139 238 238 238 F F F 8 160-179 133 133 133 228 228 228 F F F 9 180-199 138 138 138 237 237 237 F F F 10 200-219 131 131 131 224 224 224 F F F 11 220-239 131 131 131 224 224 224 F F F 12 240-259 138 138 138 237 237 237 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 137 137 137 234 234 234 F F F 15 300-319 139 139 139 238 238 238 F F F 16 320-339 138 138 138 236 236 236 F F F 17 340-359 138 138 138 237 237 237 F F F 18 360-379 139 139 139 238 238 238 F F F 19 380-399 139 139 139 238 238 238 F F F 20 400-419 134 134 134 229 229 229 F F F 21 420-439 137 137 137 234 234 234 F F F 22 440-459 134 134 134 229 229 229 F F F 23 460-479 134 134 134 229 229 229 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 137 137 137 234 234 234 F F F 27 540-559 132 132 132 225 225 225 F F F 28 560-579 139 139 139 238 238 238 F F F 29 580-599 134 134 134 229 229 229 F F F Max gen: 139, Max flush: 238 Begin loop 70 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.253453125Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.298116463Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.298802863Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.298860828Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.298881496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.298908544Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766} {"msg":"Crucible 43e00ff5-7eb4-4d19-a8ed-0d801f133591 has session id: f077d457-eb84-4185-b4cf-4b9e04d0c960","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299356945Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Upstairs opts: Upstairs UUID: 43e00ff5-7eb4-4d19-a8ed-0d801f133591, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.29941095Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Crucible stats registered with UUID: 43e00ff5-7eb4-4d19-a8ed-0d801f133591","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299432448Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:44.299448857Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299626991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"io task","client":"0","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299676407Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299708594Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"io task","client":"2","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299864001Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"task":"crutest"} {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299895958Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766} {"msg":"43e00ff5-7eb4-4d19-a8ed-0d801f133591 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299924436Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299940984Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"client":"0","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299958953Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"client":"1","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299976231Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"client":"2","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.299993099Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"io task","client":"2","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.300010408Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"io task","client":"0","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.300113969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"ds_connection connected from Ok(127.0.0.1:63347)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.30021657Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"io task","client":"0","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"ds_connection connected from Ok(127.0.0.1:53429)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.300255487Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"io task","client":"2","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"ds_connection connected from Ok(127.0.0.1:52342)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.300404654Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.301507467Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"client":"0","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.301634687Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"client":"2","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.302156851Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.302229615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.302504171Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"client":"1","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"[0]R flush_numbers[0..12]: [230, 238, 221, 234, 238, 238, 229, 238, 228, 237, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.302992428Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"[0]R generation[0..12]: [136, 139, 129, 137, 139, 139, 134, 139, 133, 138, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303022646Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303043904Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"[1]R flush_numbers[0..12]: [230, 238, 221, 234, 238, 238, 229, 238, 228, 237, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303063252Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"[1]R generation[0..12]: [136, 139, 129, 137, 139, 139, 134, 139, 133, 138, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.30308253Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303102599Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"[2]R flush_numbers[0..12]: [230, 238, 221, 234, 238, 238, 229, 238, 228, 237, 224, 224]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303121737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"[2]R generation[0..12]: [136, 139, 129, 137, 139, 139, 134, 139, 133, 138, 131, 131]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303141385Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303160994Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Wait for a query_work_queue command to finish before sending IO Max found gen is 140","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303183012Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.30320017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Next flush: 239","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303216509Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303232737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"":"downstairs","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303253455Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303269184Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"client":"0","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303286192Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"client":"1","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303303011Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"client":"2","":"downstairs","session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.30331995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303336168Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"43e00ff5-7eb4-4d19-a8ed-0d801f133591 is now active with session: f077d457-eb84-4185-b4cf-4b9e04d0c960","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303354147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303372815Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766,"session_id":"f077d457-eb84-4185-b4cf-4b9e04d0c960"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303389303Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766} {"msg":"Activated sub_volume 43e00ff5-7eb4-4d19-a8ed-0d801f133591","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:44.303404732Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 69 len 28672 data: 14 12 11 12 11 11 13 02/30 Read block 488 len 24576 03/30 Read block 86 len 16384 04/30 Read block 166 len 20480 05/30 Write block 464 len 24576 data: 11 11 9 9 9 9 06/30 Write block 291 len 28672 data: 11 10 8 12 11 9 8 07/30 Read block 302 len 4096 08/30 Write block 75 len 4096 data: 14 09/30 Write block 57 len 16384 data: 6 8 7 5 10/30 Read block 99 len 32768 11/30 Read block 560 len 4096 12/30 Read block 304 len 28672 13/30 Read block 320 len 16384 14/30 Read block 562 len 28672 15/30 Flush 16/30 Write block 403 len 20480 data: 12 10 11 13 13 17/30 Write block 404 len 8192 data: 11 12 18/30 Read block 20 len 36864 19/30 Read block 200 len 12288 20/30 Read block 405 len 28672 21/30 Read block 470 len 40960 22/30 Read block 402 len 32768 23/30 Flush 24/30 Write block 229 len 32768 data: 8 10 11 13 10 8 7 6 25/30 Write block 14 len 28672 data: 9 10 9 9 10 11 8 26/30 Read block 336 len 24576 27/30 Write block 240 len 28672 data: 9 9 9 10 7 7 10 28/30 Read block 370 len 16384 29/30 Read block 297 len 12288 30/30 Write block 318 len 16384 data: 7 8 9 11 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:15 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1015 Acked Write 5 Done Done Done false 1016 Acked Write 2 Done Done Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Read 10 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 13 1 15 Done 14 2 14 30 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1014)) Some(JobId(1022)) Downstairs last five completed: 1014 1013 1012 1011 1010 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:44.323415898Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1766} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 230 230 230 T F T <--- 1 020-039 139 139 139 238 238 238 T F T <--- 2 040-059 140 140 140 239 239 239 F F F 3 060-079 140 140 140 239 239 239 F F F 4 080-099 139 139 139 238 238 238 F F F 5 100-119 139 139 139 238 238 238 F F F 6 120-139 134 134 134 229 229 229 F F F 7 140-159 139 139 139 238 238 238 F F F 8 160-179 133 133 133 228 228 228 F F F 9 180-199 138 138 138 237 237 237 F F F 10 200-219 131 131 131 224 224 224 F F F 11 220-239 131 131 131 224 224 224 T F T <--- 12 240-259 138 138 138 237 237 237 T F T <--- 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 140 140 239 239 239 F F F 15 300-319 139 139 139 238 238 238 T F T <--- 16 320-339 138 138 138 236 236 236 T F T <--- 17 340-359 138 138 138 237 237 237 F F F 18 360-379 139 139 139 238 238 238 F F F 19 380-399 139 139 139 238 238 238 F F F 20 400-419 140 134 140 240 229 240 F T F <--- 21 420-439 137 137 137 234 234 234 F F F 22 440-459 134 134 134 229 229 229 F F F 23 460-479 140 140 140 239 239 239 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 137 137 137 234 234 234 F F F 27 540-559 132 132 132 225 225 225 F F F 28 560-579 139 139 139 238 238 238 F F F 29 580-599 134 134 134 229 229 229 F F F Max gen: 140, Max flush: 240 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 241 241 241 F F F 1 020-039 141 141 141 241 241 241 F F F 2 040-059 140 140 140 239 239 239 F F F 3 060-079 140 140 140 239 239 239 F F F 4 080-099 139 139 139 238 238 238 F F F 5 100-119 139 139 139 238 238 238 F F F 6 120-139 134 134 134 229 229 229 F F F 7 140-159 139 139 139 238 238 238 F F F 8 160-179 133 133 133 228 228 228 F F F 9 180-199 138 138 138 237 237 237 F F F 10 200-219 131 131 131 224 224 224 F F F 11 220-239 141 141 141 241 241 241 F F F 12 240-259 141 141 141 241 241 241 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 140 140 239 239 239 F F F 15 300-319 141 141 141 241 241 241 F F F 16 320-339 141 141 141 241 241 241 F F F 17 340-359 138 138 138 237 237 237 F F F 18 360-379 139 139 139 238 238 238 F F F 19 380-399 139 139 139 238 238 238 F F F 20 400-419 140 140 140 240 240 240 F F F 21 420-439 137 137 137 234 234 234 F F F 22 440-459 134 134 134 229 229 229 F F F 23 460-479 140 140 140 239 239 239 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 137 137 137 234 234 234 F F F 27 540-559 132 132 132 225 225 225 F F F 28 560-579 139 139 139 238 238 238 F F F 29 580-599 134 134 134 229 229 229 F F F Max gen: 141, Max flush: 241 Begin loop 71 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.709213077Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.753743996Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.75450651Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.754553395Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.754569684Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.754591402Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774} {"msg":"Crucible 48e5c328-2eee-4bb2-a047-82af650a939b has session id: a7fd537c-935f-4108-bb0e-cfa8133ca00c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755120556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Upstairs opts: Upstairs UUID: 48e5c328-2eee-4bb2-a047-82af650a939b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755181711Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Crucible stats registered with UUID: 48e5c328-2eee-4bb2-a047-82af650a939b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755217508Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:48.755245685Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755425859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"io task","client":"2","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755469866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755490884Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"io task","client":"1","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755642971Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"task":"crutest"} {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755665598Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774} {"msg":"48e5c328-2eee-4bb2-a047-82af650a939b active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755742342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.75576232Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"client":"0","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755780608Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"client":"1","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755797727Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"client":"2","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755814316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755831734Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"io task","client":"2","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.755848833Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"io task","client":"1","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"ds_connection connected from Ok(127.0.0.1:41767)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.756056435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"io task","client":"2","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"ds_connection connected from Ok(127.0.0.1:55410)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.75610092Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"io task","client":"1","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"ds_connection connected from Ok(127.0.0.1:41175)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.756138357Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.757374989Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"client":"2","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.757510998Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"client":"0","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.758050301Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.758208687Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.758347345Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"client":"1","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"[0]R flush_numbers[0..12]: [241, 241, 239, 239, 238, 238, 229, 238, 228, 237, 224, 241]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.758839342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"[0]R generation[0..12]: [141, 141, 140, 140, 139, 139, 134, 139, 133, 138, 131, 141]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.75886462Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.758885078Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"[1]R flush_numbers[0..12]: [241, 241, 239, 239, 238, 238, 229, 238, 228, 237, 224, 241]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.758904406Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"[1]R generation[0..12]: [141, 141, 140, 140, 139, 139, 134, 139, 133, 138, 131, 141]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.758923194Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.758943013Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"[2]R flush_numbers[0..12]: [241, 241, 239, 239, 238, 238, 229, 238, 228, 237, 224, 241]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.758961971Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"[2]R generation[0..12]: [141, 141, 140, 140, 139, 139, 134, 139, 133, 138, 131, 141]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.758980829Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759017986Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Max found gen is 142","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759043374Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759075411Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Next flush: 242","v":0,"name":"crucible","level":30,Wait for a query_work_queue command to finish before sending IO "time":"2025-12-24T00:21:48.759096419Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759134346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"":"downstairs","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759164144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759191781Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"client":"0","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759220559Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"client":"1","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759249746Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"client":"2","":"downstairs","session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759273764Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759301172Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"48e5c328-2eee-4bb2-a047-82af650a939b is now active with session: a7fd537c-935f-4108-bb0e-cfa8133ca00c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759331279Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:21:48.759356387Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774,"session_id":"a7fd537c-935f-4108-bb0e-cfa8133ca00c"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759389014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774} {"msg":"Activated sub_volume 48e5c328-2eee-4bb2-a047-82af650a939b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:48.759405743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 547 len 40960 data: 8 11 11 12 11 10 11 11 10 10 02/30 Write block 150 len 20480 data: 14 14 12 12 10 03/30 Write block 396 len 8192 data: 12 13 04/30 Read block 337 len 28672 05/30 Read block 552 len 24576 06/30 Write block 45 len 28672 data: 7 8 7 6 7 6 6 07/30 Write block 237 len 4096 data: 5 08/30 Read block 429 len 16384 09/30 Read block 31 len 32768 10/30 Read block 289 len 12288 11/30 Read block 51 len 36864 12/30 Read block 480 len 12288 13/30 Read block 132 len 12288 14/30 Write block 255 len 8192 data: 11 13 15/30 Write block 117 len 28672 data: 12 13 11 11 13 13 11 16/30 Read block 462 len 8192 17/30 Read block 409 len 16384 18/30 Read block 430 len 24576 19/30 Read block 350 len 28672 20/30 Write block 596 len 16384 data: 14 11 10 6 21/30 Read block 350 len 4096 22/30 Read block 264 len 20480 23/30 Write block 42 len 28672 data: 7 9 7 8 9 8 7 24/30 Read block 365 len 16384 25/30 Write block 124 len 16384 data: 9 8 7 10 26/30 Write block 323 len 32768 data: 11 11 10 11 11 12 11 10 27/30 Write block 217 len 12288 data: 11 13 14 28/30 Write block 202 len 16384 data: 13 13 11 10 29/30 Read block 539 len 20480 30/30 Write block 203 len 40960 data: 14 12 11 10 10 9 10 9 10 9 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Write 2 Done Sent Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Read 6 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Read 3 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Write 2 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Read 5 Done Sent Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 28 1 30 Done 29 2 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:48.777623025Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1774} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 241 241 241 F F F 1 020-039 141 141 141 241 241 241 F F F 2 040-059 140 140 140 239 239 239 T F T <--- 3 060-079 140 140 140 239 239 239 F F F 4 080-099 139 139 139 238 238 238 F F F 5 100-119 139 139 139 238 238 238 T F T <--- 6 120-139 134 134 134 229 229 229 T F T <--- 7 140-159 139 139 139 238 238 238 T T T 8 160-179 133 133 133 228 228 228 F F F 9 180-199 138 138 138 237 237 237 F F F 10 200-219 131 131 131 224 224 224 T F T <--- 11 220-239 141 141 141 241 241 241 T F T <--- 12 240-259 141 141 141 241 241 241 T F T <--- 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 140 140 239 239 239 F F F 15 300-319 141 141 141 241 241 241 F F F 16 320-339 141 141 141 241 241 241 T F T <--- 17 340-359 138 138 138 237 237 237 F F F 18 360-379 139 139 139 238 238 238 F F F 19 380-399 139 139 139 238 238 238 T F T <--- 20 400-419 140 140 140 240 240 240 F F F 21 420-439 137 137 137 234 234 234 F F F 22 440-459 134 134 134 229 229 229 F F F 23 460-479 140 140 140 239 239 239 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 137 137 137 234 234 234 F F F 27 540-559 132 132 132 225 225 225 T T T 28 560-579 139 139 139 238 238 238 F F F 29 580-599 134 134 134 229 229 229 T F T <--- Max gen: 141, Max flush: 241 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 241 241 241 F F F 1 020-039 141 141 141 241 241 241 F F F 2 040-059 142 142 142 242 242 242 F F F 3 060-079 140 140 140 239 239 239 F F F 4 080-099 139 139 139 238 238 238 F F F 5 100-119 142 142 142 242 242 242 F F F 6 120-139 142 142 142 242 242 242 F F F 7 140-159 142 142 142 242 242 242 F F F 8 160-179 133 133 133 228 228 228 F F F 9 180-199 138 138 138 237 237 237 F F F 10 200-219 142 142 142 242 242 242 F F F 11 220-239 142 142 142 242 242 242 F F F 12 240-259 142 142 142 242 242 242 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 140 140 239 239 239 F F F 15 300-319 141 141 141 241 241 241 F F F 16 320-339 142 142 142 242 242 242 F F F 17 340-359 138 138 138 237 237 237 F F F 18 360-379 139 139 139 238 238 238 F F F 19 380-399 142 142 142 242 242 242 F F F 20 400-419 140 140 140 240 240 240 F F F 21 420-439 137 137 137 234 234 234 F F F 22 440-459 134 134 134 229 229 229 F F F 23 460-479 140 140 140 239 239 239 F F F 24 480-499 131 131 131 224 224 224 F F F 25 500-519 129 129 129 221 221 221 F F F 26 520-539 137 137 137 234 234 234 F F F 27 540-559 142 142 142 242 242 242 F F F 28 560-579 139 139 139 238 238 238 F F F 29 580-599 142 142 142 242 242 242 F F F Max gen: 142, Max flush: 242 Begin loop 72 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.18513358Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.229810237Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.230898563Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.230950458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.230969626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.230996564Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782} {"msg":"Crucible 3e57e1d8-97ca-4861-b559-825750d73e16 has session id: a06f9858-ffb3-4e67-9b32-514fe9128a1d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.231587392Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Upstairs opts: Upstairs UUID: 3e57e1d8-97ca-4861-b559-825750d73e16, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.231648937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Crucible stats registered with UUID: 3e57e1d8-97ca-4861-b559-825750d73e16","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.231680544Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:53.231700553Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.231860099Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"0","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.231892146Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.231910364Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232058721Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"task":"crutest"} {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.2320785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782} {"msg":"3e57e1d8-97ca-4861-b559-825750d73e16 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232144474Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232166412Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"client":"0","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.23218414Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232201279Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"client":"2","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232217688Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232244415Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"0","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232280962Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"ds_connection connected from Ok(127.0.0.1:47774)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232324638Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"ds_connection connected from Ok(127.0.0.1:64566)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232433309Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"0","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:53.232569697Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:53.232621932Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:21:53.232657769Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232750821Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.232802917Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.233608226Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"client":"0","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.233707858Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"client":"2","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.234195165Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:21:53.234268319Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.233385403Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"ds_connection connected from Ok(127.0.0.1:60101)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.233462826Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.23502817Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"[0]R flush_numbers[0..12]: [241, 241, 242, 239, 238, 242, 242, 242, 228, 237, 242, 242]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235613579Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"[0]R generation[0..12]: [141, 141, 142, 140, 139, 142, 142, 142, 133, 138, 142, 142]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235640316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235661474Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"[1]R flush_numbers[0..12]: [241, 241, 242, 239, 238, 242, 242, 242, 228, 237, 242, 242]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235680822Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"[1]R generation[0..12]: [141, 141, 142, 140, 139, 142, 142, 142, 133, 138, 142, 142]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235700151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235721279Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"[2]R flush_numbers[0..12]: [241, 241, 242, 239, 238, 242, 242, 242, 228, 237, 242, 242]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235740657Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"[2]R generation[0..12]: [141, 141, 142, 140, 139, 142, 142, 142, 133, 138, 142, 142]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235759496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235783484Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Max found gen is 143","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235800102Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Generation requested: 144 >= found:143","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235817281Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Next flush: 243","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235833569Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235857237Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"":"downstairs","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235874136Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235889604Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"client":"0","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235915492Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"client":"1","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.2359347Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"client":"2","":"downstairs","session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235952439Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235968668Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"3e57e1d8-97ca-4861-b559-825750d73e16 is now active with session: a06f9858-ffb3-4e67-9b32-514fe9128a1d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.235986586Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.236006014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782,"session_id":"a06f9858-ffb3-4e67-9b32-514fe9128a1d"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.236029212Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782} {"msg":"Activated sub_volume 3e57e1d8-97ca-4861-b559-825750d73e16","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:03.236043921Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 98 len 36864 data: 16 13 11 12 7 8 6 9 10 02/30 Write block 348 len 24576 data: 11 11 11 8 11 12 03/30 Read block 431 len 12288 04/30 Write block 178 len 40960 data: 10 9 9 9 10 12 9 9 9 13 05/30 Write block 491 len 40960 data: 11 11 9 9 10 11 8 7 9 9 06/30 Read block 230 len 8192 07/30 Write block 530 len 8192 data: 16 14 08/30 Read block 183 len 8192 09/30 Write block 17 len 20480 data: 10 11 12 9 10 10/30 Read block 356 len 12288 11/30 Read block 333 len 20480 12/30 Read block 187 len 20480 13/30 Write block 301 len 40960 data: 13 15 15 15 17 16 15 13 10 13 14/30 Write block 43 len 40960 data: 10 8 9 10 9 8 8 7 7 4 15/30 Read block 386 len 28672 16/30 Read block 170 len 20480 17/30 Read block 18 len 12288 18/30 Write block 205 len 24576 data: 12 11 11 10 11 10 19/30 Flush 20/30 Write block 588 len 4096 data: 8 21/30 Write block 555 len 28672 data: 11 11 6 4 4 7 4 22/30 Write block 89 len 28672 data: 6 8 10 11 12 13 15 23/30 Flush 24/30 Write block 444 len 8192 data: 14 13 25/30 Read block 114 len 16384 26/30 Read block 217 len 40960 27/30 Write block 511 len 32768 data: 12 12 11 9 6 7 6 5 28/30 Read block 207 len 40960 29/30 Write block 132 len 32768 data: 11 11 10 13 11 11 12 10 30/30 Write block 312 len 28672 data: 11 10 10 10 10 10 8 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 10 Done Done Done false 1004 Acked Write 10 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Read 2 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Read 5 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 2 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 26 2 30 Done 28 4 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:03.257265192Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1782} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 144 141 144 243 241 243 F F F <--- 1 020-039 144 141 144 243 241 243 F F F <--- 2 040-059 144 142 144 243 242 243 F F F <--- 3 060-079 140 140 140 239 239 239 F F F 4 080-099 144 139 144 244 238 244 F T F <--- 5 100-119 144 142 144 243 242 243 F T F <--- 6 120-139 142 142 142 242 242 242 T F T <--- 7 140-159 142 142 142 242 242 242 F F F 8 160-179 144 133 144 243 228 243 F T F <--- 9 180-199 144 138 144 243 237 243 F T F <--- 10 200-219 144 142 144 243 242 243 F F F <--- 11 220-239 142 142 142 242 242 242 F F F 12 240-259 142 142 142 242 242 242 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 140 140 239 239 239 F F F 15 300-319 144 141 144 243 241 243 T F T <--- 16 320-339 142 142 142 242 242 242 F F F 17 340-359 144 138 144 243 237 243 F T F <--- 18 360-379 139 139 139 238 238 238 F F F 19 380-399 142 142 142 242 242 242 F F F 20 400-419 140 140 140 240 240 240 F F F 21 420-439 137 137 137 234 234 234 F F F 22 440-459 134 134 134 229 229 229 T F T <--- 23 460-479 140 140 140 239 239 239 F F F 24 480-499 144 131 144 243 224 243 F F F <--- 25 500-519 144 129 144 243 221 243 T F T <--- 26 520-539 144 137 144 243 234 243 F F F <--- 27 540-559 144 142 144 244 242 244 F F F <--- 28 560-579 144 139 144 244 238 244 F F F <--- 29 580-599 144 142 144 244 242 244 F F F <--- Max gen: 144, Max flush: 244 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 72 Verifying data now No change after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 144 144 144 243 243 243 F F F 1 020-039 144 144 144 243 243 243 F F F 2 040-059 144 144 144 243 243 243 F F F 3 060-079 140 140 140 239 239 239 F F F 4 080-099 144 144 144 244 244 244 F F F 5 100-119 144 144 144 243 243 243 F F F 6 120-139 145 145 145 245 245 245 F F F 7 140-159 142 142 142 242 242 242 F F F 8 160-179 144 144 144 243 243 243 F F F 9 180-199 144 144 144 243 243 243 F F F 10 200-219 144 144 144 243 243 243 F F F 11 220-239 142 142 142 242 242 242 F F F 12 240-259 142 142 142 242 242 242 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 140 140 239 239 239 F F F 15 300-319 145 145 145 245 245 245 F F F 16 320-339 142 142 142 242 242 242 F F F 17 340-359 144 144 144 243 243 243 F F F 18 360-379 139 139 139 238 238 238 F F F 19 380-399 142 142 142 242 242 242 F F F 20 400-419 140 140 140 240 240 240 F F F 21 420-439 137 137 137 234 234 234 F F F 22 440-459 145 145 145 245 245 245 F F F 23 460-479 140 140 140 239 239 239 F F F 24 480-499 144 144 144 243 243 243 F F F 25 500-519 145 145 145 245 245 245 F F F 26 520-539 144 144 144 243 243 243 F F F 27 540-559 144 144 144 244 244 244 F F F 28 560-579 144 144 144 244 244 244 F F F 29 580-599 144 144 144 244 244 244 F F F Max gen: 145, Max flush: 245 Begin loop 73 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.693222888Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.737972809Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.738708454Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.73875637Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.738772949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.738794807Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790} {"msg":"Crucible 9af3292a-1b36-4fcb-abef-a3ced8350406 has session id: c2929497-6036-4f25-afbb-a8d8d21b403c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.739279535Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Upstairs opts: Upstairs UUID: 9af3292a-1b36-4fcb-abef-a3ced8350406, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.739337219Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Crucible stats registered with UUID: 9af3292a-1b36-4fcb-abef-a3ced8350406","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.739371767Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:07.739397634Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.739621105Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"2","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.739658622Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.73967739Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.73978782Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"task":"crutest"} {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.739820887Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790} {"msg":"9af3292a-1b36-4fcb-abef-a3ced8350406 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.739885582Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.739918739Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"client":"0","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.739951576Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.739983713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"client":"2","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.740014191Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"2","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.740049217Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.74025127Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"ds_connection connected from Ok(127.0.0.1:34892)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.740297336Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"2","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"ds_connection connected from Ok(127.0.0.1:41402)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.740331393Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:07.74036642Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:07.740390738Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:07.740417325Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.740448813Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.740490179Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.74128884Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"client":"0","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.741376342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"client":"2","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.741899986Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:07.742015266Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.741846418Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"ds_connection connected from Ok(127.0.0.1:49292)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.741978466Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.743630792Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"[0]R flush_numbers[0..12]: [243, 243, 243, 239, 244, 243, 245, 242, 243, 243, 243, 242]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744316512Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"[0]R generation[0..12]: [144, 144, 144, 140, 144, 144, 145, 142, 144, 144, 144, 142]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.74435166Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744373838Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"[1]R flush_numbers[0..12]: [243, 243, 243, 239, 244, 243, 245, 242, 243, 243, 243, 242]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744393376Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"[1]R generation[0..12]: [144, 144, 144, 140, 144, 144, 145, 142, 144, 144, 144, 142]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744412434Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744432142Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"[2]R flush_numbers[0..12]: [243, 243, 243, 239, 244, 243, 245, 242, 243, 243, 243, 242]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744451451Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"[2]R generation[0..12]: [144, 144, 144, 140, 144, 144, 145, 142, 144, 144, 144, 142]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744474929Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744494907Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Max found gen is 146","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744511435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Generation requested: 146 >= found:146","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:22:17.744528494Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Next flush: 246","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744549932Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744565991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"":"downstairs","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744583249Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744598788Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"client":"0","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744616036Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"client":"1","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744633135Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"client":"2","":"downstairs","session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744650303Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744666362Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"9af3292a-1b36-4fcb-abef-a3ced8350406 is now active with session: c2929497-6036-4f25-afbb-a8d8d21b403c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.74468406Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744719068Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790,"session_id":"c2929497-6036-4f25-afbb-a8d8d21b403c"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744735896Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790} {"msg":"Activated sub_volume 9af3292a-1b36-4fcb-abef-a3ced8350406","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:17.744749765Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Flush 03/30 Write block 83 len 4096 data: 15 04/30 Read block 504 len 20480 05/30 Read block 184 len 36864 06/30 Read block 410 len 20480 07/30 Read block 278 len 36864 08/30 Read block 144 len 28672 09/30 Read block 496 len 8192 10/30 Read block 71 len 36864 11/30 Read block 380 len 36864 12/30 Read block 346 len 16384 13/30 Write block 243 len 40960 data: 11 8 8 11 11 10 11 13 12 12 14/30 Write block 457 len 32768 data: 10 10 11 9 9 10 10 12 15/30 Write block 359 len 12288 data: 13 14 15 16/30 Write block 151 len 4096 data: 15 17/30 Write block 427 len 36864 data: 14 13 12 13 15 16 15 14 15 18/30 Write block 349 len 8192 data: 12 12 19/30 Read block 72 len 36864 20/30 Write block 540 len 28672 data: 12 12 14 11 10 9 9 21/30 Write block 506 len 12288 data: 10 8 9 22/30 Read block 363 len 36864 23/30 Flush 24/30 Write block 583 len 24576 data: 7 7 7 7 10 9 25/30 Write block 572 len 20480 data: 9 10 10 9 9 26/30 Read block 308 len 4096 27/30 Read block 7 len 20480 28/30 Write block 65 len 40960 data: 11 12 13 14 15 13 12 13 12 12 29/30 Write block 35 len 20480 data: 9 10 10 10 10 30/30 Write block 491 len 28672 data: 12 12 10 10 11 12 9 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 1 Done Done Done false 1003 Acked Read 5 Done Sent Done false 1004 Acked Read 9 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Read 9 Done Sent Done false 1007 Acked Read 7 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Read 9 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 6 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 10 Done Sent Sent false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 27 3 32 Done 26 1 25 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1001)) Some(JobId(1022)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:17.763592663Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1790} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 144 144 144 243 243 243 F F F 1 020-039 144 144 144 243 243 243 T F T <--- 2 040-059 144 144 144 243 243 243 F F F 3 060-079 140 140 140 239 239 239 T F T <--- 4 080-099 146 144 146 248 244 248 F T F <--- 5 100-119 144 144 144 243 243 243 F F F 6 120-139 145 145 145 245 245 245 F F F 7 140-159 146 142 146 248 242 248 F F F <--- 8 160-179 144 144 144 243 243 243 F F F 9 180-199 144 144 144 243 243 243 F F F 10 200-219 144 144 144 243 243 243 F F F 11 220-239 142 142 142 242 242 242 F F F 12 240-259 146 142 146 248 242 248 F F F <--- 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 140 140 239 239 239 F F F 15 300-319 145 145 145 245 245 245 F F F 16 320-339 142 142 142 242 242 242 F F F 17 340-359 146 144 146 248 243 248 F F F <--- 18 360-379 146 139 146 248 238 248 F F F <--- 19 380-399 142 142 142 242 242 242 F F F 20 400-419 140 140 140 240 240 240 F F F 21 420-439 146 137 146 248 234 248 F F F <--- 22 440-459 146 145 146 248 245 248 F F F <--- 23 460-479 146 140 146 248 239 248 F F F <--- 24 480-499 144 144 144 243 243 243 T F T <--- 25 500-519 146 145 146 248 245 248 F F F <--- 26 520-539 144 144 144 243 243 243 F F F 27 540-559 146 144 146 248 244 248 F F F <--- 28 560-579 144 144 144 244 244 244 T F T <--- 29 580-599 144 144 144 244 244 244 T F T <--- Max gen: 146, Max flush: 248 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 73 Verifying data now No change after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 144 144 144 243 243 243 F F F 1 020-039 147 147 147 249 249 249 F F F 2 040-059 144 144 144 243 243 243 F F F 3 060-079 147 147 147 249 249 249 F F F 4 080-099 146 146 146 248 248 248 F F F 5 100-119 144 144 144 243 243 243 F F F 6 120-139 145 145 145 245 245 245 F F F 7 140-159 146 146 146 248 248 248 F F F 8 160-179 144 144 144 243 243 243 F F F 9 180-199 144 144 144 243 243 243 F F F 10 200-219 144 144 144 243 243 243 F F F 11 220-239 142 142 142 242 242 242 F F F 12 240-259 146 146 146 248 248 248 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 140 140 239 239 239 F F F 15 300-319 145 145 145 245 245 245 F F F 16 320-339 142 142 142 242 242 242 F F F 17 340-359 146 146 146 248 248 248 F F F 18 360-379 146 146 146 248 248 248 F F F 19 380-399 142 142 142 242 242 242 F F F 20 400-419 140 140 140 240 240 240 F F F 21 420-439 146 146 146 248 248 248 F F F 22 440-459 146 146 146 248 248 248 F F F 23 460-479 146 146 146 248 248 248 F F F 24 480-499 147 147 147 249 249 249 F F F 25 500-519 146 146 146 248 248 248 F F F 26 520-539 144 144 144 243 243 243 F F F 27 540-559 146 146 146 248 248 248 F F F 28 560-579 147 147 147 249 249 249 F F F 29 580-599 147 147 147 249 249 249 F F F Max gen: 147, Max flush: 249 Begin loop 74 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.18347209Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:22.228470219Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.228882733Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.260040008Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.260735047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.260780573Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.260796562Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.26081872Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798} {"msg":"Crucible 04a572c9-67fe-4d37-b920-62537ec3f3fe has session id: 44319951-2831-4b7e-83ae-a32884a50724","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.261370832Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Upstairs opts: Upstairs UUID: 04a572c9-67fe-4d37-b920-62537ec3f3fe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.261413618Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Crucible stats registered with UUID: 04a572c9-67fe-4d37-b920-62537ec3f3fe","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.261433966Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:22.261451115Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.261670386Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"io task","client":"1","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.261712172Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"io task","client":"0","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.26173268Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.261877788Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"task":"crutest"} {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.261898066Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798} {"msg":"04a572c9-67fe-4d37-b920-62537ec3f3fe active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.261974249Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.261994157Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"client":"0","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.262013665Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"client":"1","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.262042683Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"client":"2","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.26207361Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.262108127Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"io task","client":"0","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.262159373Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"io task","client":"1","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"ds_connection connected from Ok(127.0.0.1:48149)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.262198109Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"ds_connection connected from Ok(127.0.0.1:51231)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.262233206Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"io task","client":"1","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"ds_connection connected from Ok(127.0.0.1:63883)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.262263174Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"io task","client":"0","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.263532733Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"client":"2","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.26356741Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"client":"0","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.263593498Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"client":"1","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264155279Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264200785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"[0]R flush_numbers[0..12]: [243, 249, 243, 249, 248, 243, 245, 248, 243, 243, 243, 242]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.2642567Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"[0]R generation[0..12]: [144, 147, 144, 147, 146, 144, 145, 146, 144, 144, 144, 142]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264279488Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264299586Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"[1]R flush_numbers[0..12]: [243, 249, 243, 249, 248, 243, 245, 248, 243, 243, 243, 242]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264318945Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"[1]R generation[0..12]: [144, 147, 144, 147, 146, 144, 145, 146, 144, 144, 144, 142]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264337943Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264357371Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"[2]R flush_numbers[0..12]: [243, 249, 243, 249, 248, 243, 245, 248, 243, 243, 243, 242]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.26437627Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"[2]R generation[0..12]: [144, 147, 144, 147, 146, 144, 145, 146, 144, 144, 144, 142]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264395048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264414756Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Max found gen is 148","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264431225Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Generation requested: 148 >= found:148","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-12-24T00:22:22.264448533Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Next flush: 250","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264470202Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.26448659Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"":"downstairs","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264506038Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264521757Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"client":"0","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264538636Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"client":"1","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264555354Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"client":"2","":"downstairs","session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264572403Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264588421Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"04a572c9-67fe-4d37-b920-62537ec3f3fe is now active with session: 44319951-2831-4b7e-83ae-a32884a50724","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.2646063Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264621598Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798,"session_id":"44319951-2831-4b7e-83ae-a32884a50724"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264637767Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798} {"msg":"Activated sub_volume 04a572c9-67fe-4d37-b920-62537ec3f3fe","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:22.264651666Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 315 len 20480 data: 11 11 11 9 9 02/30 Write block 340 len 8192 data: 12 10 03/30 Read block 332 len 40960 04/30 Write block 228 len 36864 data: 10 9 11 12 14 11 9 8 7 05/30 Read block 451 len 24576 06/30 Write block 471 len 4096 data: 12 07/30 Read block 344 len 32768 08/30 Read block 474 len 4096 09/30 Read block 122 len 32768 10/30 Read block 327 len 16384 11/30 Flush 12/30 Flush 13/30 Write block 191 len 24576 data: 11 13 14 13 13 16 14/30 Flush 15/30 Write block 428 len 24576 data: 14 13 14 16 17 16 16/30 Write block 234 len 20480 data: 10 9 8 6 7 17/30 Flush 18/30 Flush 19/30 Write block 386 len 24576 data: 9 8 10 7 8 7 20/30 Write block 389 len 28672 data: 8 9 8 7 5 5 8 21/30 Read block 118 len 32768 22/30 Write block 11 len 20480 data: 9 10 11 10 11 23/30 Read block 578 len 24576 24/30 Read block 529 len 12288 25/30 Read block 293 len 40960 26/30 Flush 27/30 Write block 219 len 8192 data: 15 15 28/30 Write block 566 len 40960 data: 9 9 11 14 13 10 10 11 11 10 29/30 Write block 35 len 28672 data: 10 11 11 11 11 10 8 30/30 Write block 492 len 16384 data: 13 11 11 12 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Sent Done Done false 1001 Acked Write 2 Sent Done Done false 1002 Acked Read 10 Sent Done Done false 1003 Acked Write 9 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 2 Sent Done Done false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 3 3 36 Done 0 27 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:22.282945181Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1798} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 144 148 148 243 255 255 F F F <--- 1 020-039 147 147 147 249 249 249 F T T <--- 2 040-059 144 144 144 243 243 243 F T T <--- 3 060-079 147 147 147 249 249 249 F F F 4 080-099 146 146 146 248 248 248 F F F 5 100-119 144 144 144 243 243 243 F F F 6 120-139 145 145 145 245 245 245 F F F 7 140-159 146 146 146 248 248 248 F F F 8 160-179 144 144 144 243 243 243 F F F 9 180-199 144 148 148 243 252 252 F F F <--- 10 200-219 144 144 144 243 243 243 F T T <--- 11 220-239 142 148 148 242 253 253 F T T <--- 12 240-259 146 146 146 248 248 248 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 140 140 239 239 239 F F F 15 300-319 145 148 148 245 250 250 F F F <--- 16 320-339 142 142 142 242 242 242 F F F 17 340-359 146 148 148 248 250 250 F F F <--- 18 360-379 146 146 146 248 248 248 F F F 19 380-399 142 148 148 242 255 255 F F F <--- 20 400-419 140 140 140 240 240 240 F F F 21 420-439 146 148 148 248 253 253 F F F <--- 22 440-459 146 146 146 248 248 248 F F F 23 460-479 146 148 148 248 250 250 F F F <--- 24 480-499 147 147 147 249 249 249 F T T <--- 25 500-519 146 146 146 248 248 248 F F F 26 520-539 144 144 144 243 243 243 F F F 27 540-559 146 146 146 248 248 248 F F F 28 560-579 147 147 147 249 249 249 F T T <--- 29 580-599 147 147 147 249 249 249 F F F Max gen: 148, Max flush: 255 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 255 255 255 F F F 1 020-039 149 149 149 256 256 256 F F F 2 040-059 149 149 149 256 256 256 F F F 3 060-079 147 147 147 249 249 249 F F F 4 080-099 146 146 146 248 248 248 F F F 5 100-119 144 144 144 243 243 243 F F F 6 120-139 145 145 145 245 245 245 F F F 7 140-159 146 146 146 248 248 248 F F F 8 160-179 144 144 144 243 243 243 F F F 9 180-199 148 148 148 252 252 252 F F F 10 200-219 149 149 149 256 256 256 F F F 11 220-239 149 149 149 256 256 256 F F F 12 240-259 146 146 146 248 248 248 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 140 140 239 239 239 F F F 15 300-319 148 148 148 250 250 250 F F F 16 320-339 142 142 142 242 242 242 F F F 17 340-359 148 148 148 250 250 250 F F F 18 360-379 146 146 146 248 248 248 F F F 19 380-399 148 148 148 255 255 255 F F F 20 400-419 140 140 140 240 240 240 F F F 21 420-439 148 148 148 253 253 253 F F F 22 440-459 146 146 146 248 248 248 F F F 23 460-479 148 148 148 250 250 250 F F F 24 480-499 149 149 149 256 256 256 F F F 25 500-519 146 146 146 248 248 248 F F F 26 520-539 144 144 144 243 243 243 F F F 27 540-559 146 146 146 248 248 248 F F F 28 560-579 149 149 149 256 256 256 F F F 29 580-599 147 147 147 249 249 249 F F F Max gen: 149, Max flush: 256 Begin loop 75 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.722378604Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:26.767350835Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.767764749Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.79873473Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.799449597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.799501473Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.799525071Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.799552488Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806} {"msg":"Crucible f931d089-f850-4b99-9581-150655cee5df has session id: 8eaa9a59-6fee-4c4a-8967-891ac80bea79","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800030157Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Upstairs opts: Upstairs UUID: f931d089-f850-4b99-9581-150655cee5df, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800095701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Crucible stats registered with UUID: f931d089-f850-4b99-9581-150655cee5df","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800119719Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:26.800136988Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800366768Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"io task","client":"2","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800411133Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800431582Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.80057073Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"task":"crutest"} {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800592268Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806} {"msg":"f931d089-f850-4b99-9581-150655cee5df active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800658132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.80067924Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"client":"0","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800696809Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"client":"1","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800713757Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"client":"2","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800734255Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"io task","client":"2","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800755623Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.800788791Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"ds_connection connected from Ok(127.0.0.1:46606)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.80102927Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"ds_connection connected from Ok(127.0.0.1:36224)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.801076726Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"ds_connection connected from Ok(127.0.0.1:50396)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.801098234Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"io task","client":"2","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.802254153Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"client":"1","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.80228939Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"client":"2","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.802310308Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"client":"0","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.802904676Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.802934684Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"[0]R flush_numbers[0..12]: [255, 256, 256, 249, 248, 243, 245, 248, 243, 252, 256, 256]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.802953772Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"[0]R generation[0..12]: [148, 149, 149, 147, 146, 144, 145, 146, 144, 148, 149, 149]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.80297341Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.802994319Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"[1]R flush_numbers[0..12]: [255, 256, 256, 249, 248, 243, 245, 248, 243, 252, 256, 256]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803028416Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"[1]R generation[0..12]: [148, 149, 149, 147, 146, 144, 145, 146, 144, 148, 149, 149]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803051303Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803071062Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"[2]R flush_numbers[0..12]: [255, 256, 256, 249, 248, 243, 245, 248, 243, 252, 256, 256]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.80308979Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"[2]R generation[0..12]: [148, 149, 149, 147, 146, 144, 145, 146, 144, 148, 149, 149]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803108669Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803133476Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Max found gen is 150","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803153734Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Generation requested: 150 >= found:150","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803171163Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Next flush: 257","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803187382Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.80320338Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"":"downstairs","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803220429Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Wait for a query_work_queue command to finish before sending IO Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803241657Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"client":"0","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803258576Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"client":"1","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803275264Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"client":"2","":"downstairs","session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803292442Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803310521Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"f931d089-f850-4b99-9581-150655cee5df is now active with session: 8eaa9a59-6fee-4c4a-8967-891ac80bea79","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803328509Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803343868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806,"session_id":"8eaa9a59-6fee-4c4a-8967-891ac80bea79"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803359757Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806} {"msg":"Activated sub_volume f931d089-f850-4b99-9581-150655cee5df","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:26.803373625Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 75 len 16384 data: 15 11 14 11 02/30 Write block 138 len 40960 data: 13 11 11 13 13 10 9 10 7 7 03/30 Read block 250 len 28672 04/30 Read block 488 len 20480 05/30 Read block 370 len 24576 06/30 Write block 169 len 16384 data: 8 11 11 10 07/30 Write block 45 len 12288 data: 10 11 10 08/30 Write block 287 len 8192 data: 10 10 09/30 Read block 179 len 4096 10/30 Write block 480 len 4096 data: 10 11/30 Write block 463 len 8192 data: 11 13 12/30 Read block 531 len 36864 13/30 Read block 343 len 8192 14/30 Write block 211 len 32768 data: 11 10 9 11 10 11 12 14 15/30 Flush 16/30 Write block 497 len 24576 data: 10 8 10 10 9 7 17/30 Read block 396 len 36864 18/30 Read block 454 len 12288 19/30 Read block 33 len 32768 20/30 Read block 215 len 16384 21/30 Write block 60 len 12288 data: 6 7 9 22/30 Write block 164 len 4096 data: 14 23/30 Read block 256 len 32768 24/30 Flush 25/30 Write block 345 len 20480 data: 7 9 11 12 13 26/30 Read block 371 len 36864 27/30 Read block 286 len 8192 28/30 Read block 387 len 12288 29/30 Read block 446 len 40960 30/30 Write block 94 len 32768 data: 14 16 14 16 17 14 12 13 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Read 5 Done Done Done false 1004 Acked Read 6 Done Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Write 1 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Read 4 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 1 1 27 Done 5 29 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:26.82191264Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1806} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 255 255 255 F F F 1 020-039 149 149 149 256 256 256 F F F 2 040-059 149 150 150 256 257 257 F F F <--- 3 060-079 147 150 150 249 258 258 T F F <--- 4 080-099 146 146 146 248 248 248 F T T <--- 5 100-119 144 144 144 243 243 243 F T T <--- 6 120-139 145 150 150 245 257 257 T F F <--- 7 140-159 146 150 150 248 257 257 T F F <--- 8 160-179 144 150 150 243 258 258 F F F <--- 9 180-199 148 148 148 252 252 252 F F F 10 200-219 149 150 150 256 257 257 F F F <--- 11 220-239 149 149 149 256 256 256 F F F 12 240-259 146 146 146 248 248 248 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 140 150 150 239 257 257 F F F <--- 15 300-319 148 148 148 250 250 250 F F F 16 320-339 142 142 142 242 242 242 F F F 17 340-359 148 148 148 250 250 250 F T T <--- 18 360-379 146 146 146 248 248 248 F F F 19 380-399 148 148 148 255 255 255 F F F 20 400-419 140 140 140 240 240 240 F F F 21 420-439 148 148 148 253 253 253 F F F 22 440-459 146 146 146 248 248 248 F F F 23 460-479 148 150 150 250 257 257 F F F <--- 24 480-499 149 150 150 256 258 258 F F F <--- 25 500-519 146 150 150 248 258 258 F F F <--- 26 520-539 144 144 144 243 243 243 F F F 27 540-559 146 146 146 248 248 248 F F F 28 560-579 149 149 149 256 256 256 F F F 29 580-599 147 147 147 249 249 249 F F F Max gen: 150, Max flush: 258 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 255 255 255 F F F 1 020-039 149 149 149 256 256 256 F F F 2 040-059 150 150 150 257 257 257 F F F 3 060-079 150 150 150 258 258 258 F F F 4 080-099 151 151 151 259 259 259 F F F 5 100-119 151 151 151 259 259 259 F F F 6 120-139 150 150 150 257 257 257 F F F 7 140-159 150 150 150 257 257 257 F F F 8 160-179 150 150 150 258 258 258 F F F 9 180-199 148 148 148 252 252 252 F F F 10 200-219 150 150 150 257 257 257 F F F 11 220-239 149 149 149 256 256 256 F F F 12 240-259 146 146 146 248 248 248 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 150 150 150 257 257 257 F F F 15 300-319 148 148 148 250 250 250 F F F 16 320-339 142 142 142 242 242 242 F F F 17 340-359 151 151 151 259 259 259 F F F 18 360-379 146 146 146 248 248 248 F F F 19 380-399 148 148 148 255 255 255 F F F 20 400-419 140 140 140 240 240 240 F F F 21 420-439 148 148 148 253 253 253 F F F 22 440-459 146 146 146 248 248 248 F F F 23 460-479 150 150 150 257 257 257 F F F 24 480-499 150 150 150 258 258 258 F F F 25 500-519 150 150 150 258 258 258 F F F 26 520-539 144 144 144 243 243 243 F F F 27 540-559 146 146 146 248 248 248 F F F 28 560-579 149 149 149 256 256 256 F F F 29 580-599 147 147 147 249 249 249 F F F Max gen: 151, Max flush: 259 Begin loop 76 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.262031524Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:31.305761793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.306162318Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.337078514Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.337737156Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.337793822Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.33781364Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.337840777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814} {"msg":"Crucible e2d1fcb0-cae8-46a8-b25f-3b23d373bfce has session id: 4527d1e0-4af2-42a0-accc-7d5e7d45a4ad","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.338454104Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Upstairs opts: Upstairs UUID: e2d1fcb0-cae8-46a8-b25f-3b23d373bfce, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.338508319Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Crucible stats registered with UUID: e2d1fcb0-cae8-46a8-b25f-3b23d373bfce","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.338532227Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:31.338549665Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.338717681Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"io task","client":"1","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.338754708Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.338786465Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"io task","client":"2","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.338912634Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"task":"crutest"} {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.338934302Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814} {"msg":"e2d1fcb0-cae8-46a8-b25f-3b23d373bfce active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.339007316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.339028124Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"client":"0","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.339046222Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"client":"1","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.339064101Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"client":"2","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.339081479Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"io task","client":"2","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.339103117Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.339130485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"io task","client":"1","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:62687)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.339316179Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"io task","client":"2","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:43758)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.339407891Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"io task","client":"1","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:43499)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.339448167Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.340535543Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"client":"2","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.34056952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"client":"1","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.340736715Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"client":"0","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341232342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341271878Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"[0]R flush_numbers[0..12]: [255, 256, 257, 258, 259, 259, 257, 257, 258, 252, 257, 256]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341293786Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"[0]R generation[0..12]: [148, 149, 150, 150, 151, 151, 150, 150, 150, 148, 150, 149]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341313775Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341333843Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"[1]R flush_numbers[0..12]: [255, 256, 257, 258, 259, 259, 257, 257, 258, 252, 257, 256]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341353411Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"[1]R generation[0..12]: [148, 149, 150, 150, 151, 151, 150, 150, 150, 148, 150, 149]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.3413723Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341392298Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"[2]R flush_numbers[0..12]: [255, 256, 257, 258, 259, 259, 257, 257, 258, 252, 257, 256]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341411486Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"[2]R generation[0..12]: [148, 149, 150, 150, 151, 151, 150, 150, 150, 148, 150, 149]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341430385Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341457082Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.3414871Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341518677Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Next flush: 260","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341541295Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341563903Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"":"downstairs","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341581201Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.34159667Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"client":"0","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341614009Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"client":"1","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341631017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"client":"2","":"downstairs","session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341648515Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341664574Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"e2d1fcb0-cae8-46a8-b25f-3b23d373bfce is now active with session: 4527d1e0-4af2-42a0-accc-7d5e7d45a4ad","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341682552Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341698081Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814,"session_id":"4527d1e0-4af2-42a0-accc-7d5e7d45a4ad"} {"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.34171409Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814} {"msg":"Activated sub_volume e2d1fcb0-cae8-46a8-b25f-3b23d373bfce","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:31.341728089Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 44 len 8192 data: 9 11 03/30 Write block 320 len 24576 data: 10 12 12 12 12 11 04/30 Read block 557 len 32768 05/30 Write block 67 len 20480 data: 14 15 16 14 13 06/30 Read block 479 len 32768 07/30 Read block 422 len 16384 08/30 Write block 391 len 4096 data: 9 09/30 Write block 477 len 16384 data: 5 5 6 11 10/30 Read block 275 len 20480 11/30 Write block 217 len 32768 data: 13 15 16 16 17 16 11 13 12/30 Read block 455 len 40960 13/30 Read block 542 len 8192 14/30 Read block 125 len 24576 15/30 Write block 530 len 16384 data: 17 15 12 12 16/30 Read block 101 len 4096 17/30 Flush 18/30 Write block 103 len 36864 data: 9 7 10 11 9 10 10 10 10 19/30 Write block 562 len 24576 data: 6 8 9 8 10 10 20/30 Read block 72 len 20480 21/30 Read block 108 len 12288 22/30 Write block 163 len 40960 data: 15 15 13 15 12 10 9 12 12 11 23/30 Read block 391 len 16384 24/30 Write block 583 len 4096 data: 8 25/30 Write block 140 len 12288 data: 12 14 14 26/30 Read block 197 len 16384 27/30 Read block 188 len 24576 28/30 Read block 27 len 24576 29/30 Read block 311 len 12288 30/30 Write block 10 len 20480 data: 11 10 11 12 11 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 2 Sent Done Done false 1002 Acked Write 6 Sent Done Done false 1003 Acked Read 8 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Read 1 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 9 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:31.361514114Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1814} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 255 255 255 F T T <--- 1 020-039 149 149 149 256 256 256 F F F 2 040-059 150 152 152 257 261 261 F F F <--- 3 060-079 150 152 152 258 261 261 F F F <--- 4 080-099 151 151 151 259 259 259 F F F 5 100-119 151 151 151 259 259 259 F T T <--- 6 120-139 150 150 150 257 257 257 F F F 7 140-159 150 150 150 257 257 257 F T T <--- 8 160-179 150 150 150 258 258 258 F T T <--- 9 180-199 148 148 148 252 252 252 F F F 10 200-219 150 152 152 257 261 261 F F F <--- 11 220-239 149 152 152 256 261 261 F F F <--- 12 240-259 146 146 146 248 248 248 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 150 150 150 257 257 257 F F F 15 300-319 148 148 148 250 250 250 F F F 16 320-339 142 152 152 242 261 261 F F F <--- 17 340-359 151 151 151 259 259 259 F F F 18 360-379 146 146 146 248 248 248 F F F 19 380-399 148 152 152 255 261 261 F F F <--- 20 400-419 140 140 140 240 240 240 F F F 21 420-439 148 148 148 253 253 253 F F F 22 440-459 146 146 146 248 248 248 F F F 23 460-479 150 152 152 257 261 261 F F F <--- 24 480-499 150 152 152 258 261 261 F F F <--- 25 500-519 150 150 150 258 258 258 F F F 26 520-539 144 152 152 243 261 261 F F F <--- 27 540-559 146 146 146 248 248 248 F F F 28 560-579 149 149 149 256 256 256 F T T <--- 29 580-599 147 147 147 249 249 249 F T T <--- Max gen: 152, Max flush: 261 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 153 153 153 262 262 262 F F F 1 020-039 149 149 149 256 256 256 F F F 2 040-059 152 152 152 261 261 261 F F F 3 060-079 152 152 152 261 261 261 F F F 4 080-099 151 151 151 259 259 259 F F F 5 100-119 153 153 153 262 262 262 F F F 6 120-139 150 150 150 257 257 257 F F F 7 140-159 153 153 153 262 262 262 F F F 8 160-179 153 153 153 262 262 262 F F F 9 180-199 148 148 148 252 252 252 F F F 10 200-219 152 152 152 261 261 261 F F F 11 220-239 152 152 152 261 261 261 F F F 12 240-259 146 146 146 248 248 248 F F F 13 260-279 139 139 139 238 238 238 F F F 14 280-299 150 150 150 257 257 257 F F F 15 300-319 148 148 148 250 250 250 F F F 16 320-339 152 152 152 261 261 261 F F F 17 340-359 151 151 151 259 259 259 F F F 18 360-379 146 146 146 248 248 248 F F F 19 380-399 152 152 152 261 261 261 F F F 20 400-419 140 140 140 240 240 240 F F F 21 420-439 148 148 148 253 253 253 F F F 22 440-459 146 146 146 248 248 248 F F F 23 460-479 152 152 152 261 261 261 F F F 24 480-499 152 152 152 261 261 261 F F F 25 500-519 150 150 150 258 258 258 F F F 26 520-539 152 152 152 261 261 261 F F F 27 540-559 146 146 146 248 248 248 F F F 28 560-579 153 153 153 262 262 262 F F F 29 580-599 153 153 153 262 262 262 F F F Max gen: 153, Max flush: 262 Begin loop 77 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.837700195Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.883284853Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.884174615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.884221571Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.88423814Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.884260058Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822} {"msg":"Crucible 5585bda5-84b1-4ecd-bf8b-ab472e362225 has session id: 76308a23-8c0f-4486-b497-14d5ef98c517","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.884796301Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Upstairs opts: Upstairs UUID: 5585bda5-84b1-4ecd-bf8b-ab472e362225, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.884858426Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Crucible stats registered with UUID: 5585bda5-84b1-4ecd-bf8b-ab472e362225","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.884884593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:35.884905561Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885104534Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"io task","client":"1","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.88514971Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885168718Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"io task","client":"0","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885272289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"task":"crutest"} {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885292338Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822} {"msg":"5585bda5-84b1-4ecd-bf8b-ab472e362225 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885350143Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.88538479Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"client":"0","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885404858Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"client":"1","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885427426Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"client":"2","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885457364Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885483591Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"io task","client":"0","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885509879Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"io task","client":"1","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"ds_connection connected from Ok(127.0.0.1:55052)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885540876Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"ds_connection connected from Ok(127.0.0.1:60681)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885572983Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"io task","client":"1","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"ds_connection connected from Ok(127.0.0.1:33327)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.885604491Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"io task","client":"0","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.88675969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"client":"1","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.886809556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"client":"0","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.886902958Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"client":"2","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887278445Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887392015Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"[0]R flush_numbers[0..12]: [262, 256, 261, 261, 259, 262, 257, 262, 262, 252, 261, 261]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887436821Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"[0]R generation[0..12]: [153, 149, 152, 152, 151, 153, 150, 153, 153, 148, 152, 152]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887461349Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887481797Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"[1]R flush_numbers[0..12]: [262, 256, 261, 261, 259, 262, 257, 262, 262, 252, 261, 261]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887501125Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"[1]R generation[0..12]: [153, 149, 152, 152, 151, 153, 150, 153, 153, 148, 152, 152]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887519844Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887539442Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"[2]R flush_numbers[0..12]: [262, 256, 261, 261, 259, 262, 257, 262, 262, 252, 261, 261]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.88755875Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"[2]R generation[0..12]: [153, 149, 152, 152, 151, 153, 150, 153, 153, 148, 152, 152]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887577819Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887597637Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Max found gen is 154","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887614145Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Generation requested: 154 >= found:154","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887631244Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Next flush: 263","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887647213Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {Wait for a query_work_queue command to finish before sending IO "msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887669301Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"":"downstairs","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887686389Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887701888Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"client":"0","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887722456Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"client":"1","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887740014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"client":"2","":"downstairs","session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887757233Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887773412Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"5585bda5-84b1-4ecd-bf8b-ab472e362225 is now active with session: 76308a23-8c0f-4486-b497-14d5ef98c517","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.88779122Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887806649Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822,"session_id":"76308a23-8c0f-4486-b497-14d5ef98c517"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887822847Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822} {"msg":"Activated sub_volume 5585bda5-84b1-4ecd-bf8b-ab472e362225","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:35.887836686Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 293 len 28672 data: 9 13 12 10 9 8 11 02/30 Write block 56 len 8192 data: 4 7 03/30 Read block 27 len 24576 04/30 Read block 533 len 16384 05/30 Write block 205 len 36864 data: 13 12 12 11 12 11 12 11 10 06/30 Read block 531 len 12288 07/30 Write block 140 len 8192 data: 13 15 08/30 Write block 80 len 16384 data: 11 11 13 16 09/30 Write block 39 len 24576 data: 12 11 9 8 11 10 10/30 Flush 11/30 Write block 347 len 36864 data: 12 13 14 13 9 12 13 14 15 12/30 Flush 13/30 Write block 439 len 40960 data: 13 13 15 14 14 15 14 13 15 14 14/30 Write block 10 len 40960 data: 12 11 12 13 12 12 10 11 12 13 15/30 Write block 31 len 36864 data: 11 8 12 8 11 12 12 12 13 16/30 Write block 285 len 8192 data: 9 10 17/30 Write block 418 len 40960 data: 9 11 13 11 9 8 9 10 12 15 18/30 Write block 276 len 40960 data: 15 13 10 8 11 13 12 12 10 10 19/30 Write block 455 len 8192 data: 12 11 20/30 Read block 566 len 24576 21/30 Read block 100 len 32768 22/30 Read block 363 len 36864 23/30 Flush 24/30 Write block 169 len 36864 data: 10 13 13 12 10 10 9 12 11 25/30 Read block 216 len 28672 26/30 Read block 191 len 32768 27/30 Read block 220 len 32768 28/30 Write block 446 len 16384 data: 14 16 15 17 29/30 Write block 256 len 28672 data: 14 15 12 12 12 12 14 30/30 Read block 70 len 12288 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 2 Done Done Sent false 1002 Acked Read 6 Done Done Sent false 1003 Acked Read 4 Done Done Sent false 1004 Acked Write 9 Done Done Sent false 1005 Acked Read 3 Done Done Sent false 1006 Acked Write 2 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 6 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 9 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Write 9 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 9 Done Done Sent false 1024 Acked Read 7 Done Done Sent false 1025 Acked Read 8 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Write 7 Done Done Sent false 1029 Acked Read 3 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 29 30 Done 29 30 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:35.908132068Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1822} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 153 265 265 262 F F F <--- 1 020-039 154 154 149 265 265 256 F F F <--- 2 040-059 154 154 152 263 263 261 F F F <--- 3 060-079 152 152 152 261 261 261 F F F 4 080-099 154 154 151 263 263 259 F F F <--- 5 100-119 153 153 153 262 262 262 F F F 6 120-139 150 150 150 257 257 257 F F F 7 140-159 154 154 153 263 263 262 F F F <--- 8 160-179 153 153 153 262 262 262 T T F <--- 9 180-199 148 148 148 252 252 252 F F F 10 200-219 154 154 152 263 263 261 F F F <--- 11 220-239 152 152 152 261 261 261 F F F 12 240-259 146 146 146 248 248 248 T T F <--- 13 260-279 154 154 139 265 265 238 T T F <--- 14 280-299 154 154 150 265 265 257 F F T <--- 15 300-319 148 148 148 250 250 250 F F F 16 320-339 152 152 152 261 261 261 F F F 17 340-359 154 154 151 264 264 259 F F F <--- 18 360-379 146 146 146 248 248 248 F F F 19 380-399 152 152 152 261 261 261 F F F 20 400-419 154 154 140 265 265 240 F F F <--- 21 420-439 154 154 148 265 265 253 F F F <--- 22 440-459 154 154 146 265 265 248 T T F <--- 23 460-479 152 152 152 261 261 261 F F F 24 480-499 152 152 152 261 261 261 F F F 25 500-519 150 150 150 258 258 258 F F F 26 520-539 152 152 152 261 261 261 F F F 27 540-559 146 146 146 248 248 248 F F F 28 560-579 153 153 153 262 262 262 F F F 29 580-599 153 153 153 262 262 262 F F F Max gen: 154, Max flush: 265 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 77 Verifying data now No change after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 154 265 265 265 F F F 1 020-039 154 154 154 265 265 265 F F F 2 040-059 154 154 154 263 263 263 F F F 3 060-079 152 152 152 261 261 261 F F F 4 080-099 154 154 154 263 263 263 F F F 5 100-119 153 153 153 262 262 262 F F F 6 120-139 150 150 150 257 257 257 F F F 7 140-159 154 154 154 263 263 263 F F F 8 160-179 155 155 155 266 266 266 F F F 9 180-199 148 148 148 252 252 252 F F F 10 200-219 154 154 154 263 263 263 F F F 11 220-239 152 152 152 261 261 261 F F F 12 240-259 155 155 155 266 266 266 F F F 13 260-279 155 155 155 266 266 266 F F F 14 280-299 154 154 154 265 265 265 F F F 15 300-319 148 148 148 250 250 250 F F F 16 320-339 152 152 152 261 261 261 F F F 17 340-359 154 154 154 264 264 264 F F F 18 360-379 146 146 146 248 248 248 F F F 19 380-399 152 152 152 261 261 261 F F F 20 400-419 154 154 154 265 265 265 F F F 21 420-439 154 154 154 265 265 265 F F F 22 440-459 155 155 155 266 266 266 F F F 23 460-479 152 152 152 261 261 261 F F F 24 480-499 152 152 152 261 261 261 F F F 25 500-519 150 150 150 258 258 258 F F F 26 520-539 152 152 152 261 261 261 F F F 27 540-559 146 146 146 248 248 248 F F F 28 560-579 153 153 153 262 262 262 F F F 29 580-599 153 153 153 262 262 262 F F F Max gen: 155, Max flush: 266 Begin loop 78 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.323959878Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.368211151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.368960496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.369008372Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.369024951Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.369046819Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830} {"msg":"Crucible e81f51d6-9ac0-4aac-aa89-24a11fb604ee has session id: b1649b2e-6472-40b7-8306-099653004cb2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.369520368Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Upstairs opts: Upstairs UUID: e81f51d6-9ac0-4aac-aa89-24a11fb604ee, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.369583242Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Crucible stats registered with UUID: e81f51d6-9ac0-4aac-aa89-24a11fb604ee","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.369612029Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:40.369631628Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.36983777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.369887596Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"io task","client":"0","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.369920043Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"io task","client":"2","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.369986497Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"task":"crutest"} {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.370025064Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830} {"msg":"e81f51d6-9ac0-4aac-aa89-24a11fb604ee active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.370083718Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.370110026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"client":"0","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.370127975Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"client":"1","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.370145533Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"client":"2","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.37017669Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"io task","client":"2","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.370195869Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"io task","client":"0","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.370213457Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"ds_connection connected from Ok(127.0.0.1:38267)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.370401911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"io task","client":"2","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"ds_connection connected from Ok(127.0.0.1:64551)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.370451136Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"ds_connection connected from Ok(127.0.0.1:40513)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.370529849Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"io task","client":"0","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.37155278Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"client":"0","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.371660801Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"client":"1","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.372166957Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.37224077Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.372488709Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"client":"2","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"[0]R flush_numbers[0..12]: [265, 265, 263, 261, 263, 262, 257, 263, 266, 252, 263, 261]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.372993585Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"[0]R generation[0..12]: [154, 154, 154, 152, 154, 153, 150, 154, 155, 148, 154, 152]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373024132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.37304961Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"[1]R flush_numbers[0..12]: [265, 265, 263, 261, 263, 262, 257, 263, 266, 252, 263, 261]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373073408Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"[1]R generation[0..12]: [154, 154, 154, 152, 154, 153, 150, 154, 155, 148, 154, 152]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373097156Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373121783Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"[2]R flush_numbers[0..12]: [265, 265, 263, 261, 263, 262, 257, 263, 266, 252, 263, 261]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373145372Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"[2]R generation[0..12]: [154, 154, 154, 152, 154, 153, 150, 154, 155, 148, 154, 152]","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-12-24T00:22:40.373174139Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373206006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373226414Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Generation requested: 156 >= found:156","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373247543Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Next flush: 267","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373267601Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373287709Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"":"downstairs","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373308927Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373327966Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"client":"0","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373349234Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"client":"1","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 } {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373374872Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"client":"2","":"downstairs","session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.3733962Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373416768Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"e81f51d6-9ac0-4aac-aa89-24a11fb604ee is now active with session: b1649b2e-6472-40b7-8306-099653004cb2","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373439006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373457904Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830,"session_id":"b1649b2e-6472-40b7-8306-099653004cb2"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373477963Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830} {"msg":"Activated sub_volume e81f51d6-9ac0-4aac-aa89-24a11fb604ee","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:40.373497631Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 105 len 28672 data: 11 12 10 11 11 11 11 02/30 Write block 63 len 4096 data: 10 03/30 Write block 394 len 8192 data: 6 9 04/30 Flush 05/30 Write block 159 len 12288 data: 12 15 19 06/30 Write block 106 len 28672 data: 13 11 12 12 12 12 9 07/30 Read block 242 len 32768 08/30 Write block 101 len 28672 data: 14 8 10 8 12 14 12 09/30 Read block 486 len 20480 10/30 Read block 256 len 20480 11/30 Write block 149 len 24576 data: 12 15 16 13 13 11 12/30 Read block 7 len 20480 13/30 Write block 264 len 32768 data: 10 11 11 12 11 13 11 13 14/30 Write block 324 len 24576 data: 13 12 12 12 13 12 15/30 Write block 482 len 24576 data: 13 14 14 11 12 12 16/30 Read block 420 len 28672 17/30 Write block 365 len 24576 data: 12 12 11 9 12 8 18/30 Write block 362 len 32768 data: 15 13 14 13 13 12 10 13 19/30 Read block 88 len 8192 20/30 Read block 14 len 32768 21/30 Write block 209 len 28672 data: 13 12 13 12 11 12 11 22/30 Write block 275 len 20480 data: 13 16 14 11 9 23/30 Write block 6 len 24576 data: 13 13 8 9 13 12 24/30 Write block 591 len 16384 data: 13 12 12 14 25/30 Read block 371 len 36864 26/30 Write block 550 len 4096 data: 13 27/30 Write block 329 len 36864 data: 13 11 12 10 10 9 9 9 10 28/30 Write block 80 len 16384 data: 12 12 14 17 29/30 Write block 363 len 32768 data: 14 15 14 14 13 11 14 9 30/30 Read block 196 len 8192 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 3 Done Done Done false 1005 Acked Write 7 Done Done Done false 1006 Acked Read 8 Done Done Done false 1007 Acked Write 7 Done Done Done false 1008 Acked Read 5 Done Done Done false 1009 Acked Read 5 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Read 5 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Read 8 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Read 9 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Write 8 Done Sent Sent false 1029 Acked Read 2 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 2 21 23 Done 26 24 5 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:40.390995676Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1830} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 154 265 265 265 T T F <--- 1 020-039 154 154 154 265 265 265 F F F 2 040-059 154 154 154 263 263 263 F F F 3 060-079 156 156 156 267 267 267 F F F 4 080-099 154 154 154 263 263 263 T T F <--- 5 100-119 156 156 156 267 267 267 T T T 6 120-139 150 150 150 257 257 257 F F F 7 140-159 154 154 154 263 263 263 T T T 8 160-179 155 155 155 266 266 266 T T T 9 180-199 148 148 148 252 252 252 F F F 10 200-219 154 154 154 263 263 263 T T F <--- 11 220-239 152 152 152 261 261 261 F F F 12 240-259 155 155 155 266 266 266 F F F 13 260-279 155 155 155 266 266 266 T T F <--- 14 280-299 154 154 154 265 265 265 F F F 15 300-319 148 148 148 250 250 250 F F F 16 320-339 152 152 152 261 261 261 T T F <--- 17 340-359 154 154 154 264 264 264 F F F 18 360-379 146 146 146 248 248 248 T T F <--- 19 380-399 156 156 156 267 267 267 F F F 20 400-419 154 154 154 265 265 265 F F F 21 420-439 154 154 154 265 265 265 F F F 22 440-459 155 155 155 266 266 266 F F F 23 460-479 152 152 152 261 261 261 F F F 24 480-499 152 152 152 261 261 261 T T F <--- 25 500-519 150 150 150 258 258 258 F F F 26 520-539 152 152 152 261 261 261 F F F 27 540-559 146 146 146 248 248 248 T T F <--- 28 560-579 153 153 153 262 262 262 F F F 29 580-599 153 153 153 262 262 262 T T F <--- Max gen: 156, Max flush: 267 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 268 268 268 F F F 1 020-039 154 154 154 265 265 265 F F F 2 040-059 154 154 154 263 263 263 F F F 3 060-079 156 156 156 267 267 267 F F F 4 080-099 157 157 157 268 268 268 F F F 5 100-119 157 157 157 268 268 268 F F F 6 120-139 150 150 150 257 257 257 F F F 7 140-159 157 157 157 268 268 268 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 148 148 148 252 252 252 F F F 10 200-219 157 157 157 268 268 268 F F F 11 220-239 152 152 152 261 261 261 F F F 12 240-259 155 155 155 266 266 266 F F F 13 260-279 157 157 157 268 268 268 F F F 14 280-299 154 154 154 265 265 265 F F F 15 300-319 148 148 148 250 250 250 F F F 16 320-339 157 157 157 268 268 268 F F F 17 340-359 154 154 154 264 264 264 F F F 18 360-379 157 157 157 268 268 268 F F F 19 380-399 156 156 156 267 267 267 F F F 20 400-419 154 154 154 265 265 265 F F F 21 420-439 154 154 154 265 265 265 F F F 22 440-459 155 155 155 266 266 266 F F F 23 460-479 152 152 152 261 261 261 F F F 24 480-499 157 157 157 268 268 268 F F F 25 500-519 150 150 150 258 258 258 F F F 26 520-539 152 152 152 261 261 261 F F F 27 540-559 157 157 157 268 268 268 F F F 28 560-579 153 153 153 262 262 262 F F F 29 580-599 157 157 157 268 268 268 F F F Max gen: 157, Max flush: 268 Begin loop 79 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.801857119Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.846318244Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.84705163Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.847099766Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.847116754Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.847138602Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838} {"msg":"Crucible 3902eb81-e138-4f85-a0d1-89320a414896 has session id: 562b8c16-ffda-454c-b47d-4ffe97cb86f0","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.847703973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Upstairs opts: Upstairs UUID: 3902eb81-e138-4f85-a0d1-89320a414896, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.847750229Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Crucible stats registered with UUID: 3902eb81-e138-4f85-a0d1-89320a414896","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.847770147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:44.847786026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.84797258Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"io task","client":"0","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848031805Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"io task","client":"1","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848053833Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848194961Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"task":"crutest"} {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848218698Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838} {"msg":"3902eb81-e138-4f85-a0d1-89320a414896 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848285422Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848305271Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"client":"0","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848322929Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"client":"1","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848339818Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"client":"2","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848357676Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848399473Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"io task","client":"0","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.84843335Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"io task","client":"1","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"ds_connection connected from Ok(127.0.0.1:46156)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848471096Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"ds_connection connected from Ok(127.0.0.1:39144)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848504583Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"io task","client":"0","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"ds_connection connected from Ok(127.0.0.1:49957)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.848537791Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"io task","client":"1","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.849902082Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"client":"0","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.850044329Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"client":"1","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.850512268Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.850600751Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851125535Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"client":"2","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"[0]R flush_numbers[0..12]: [268, 265, 263, 267, 268, 268, 257, 268, 268, 252, 268, 261]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851610593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"[0]R generation[0..12]: [157, 154, 154, 156, 157, 157, 150, 157, 157, 148, 157, 152]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.85164674Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851672647Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"[1]R flush_numbers[0..12]: [268, 265, 263, 267, 268, 268, 257, 268, 268, 252, 268, 261]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851696605Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"[1]R generation[0..12]: [157, 154, 154, 156, 157, 157, 150, 157, 157, 148, 157, 152]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851720313Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851744771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"[2]R flush_numbers[0..12]: [268, 265, 263, 267, 268, 268, 257, 268, 268, 252, 268, 261]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851768239Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"[2]R generation[0..12]: [157, 154, 154, 156, 157, 157, 150, 157, 157, 148, 157, 152]Wait for a query_work_queue command to finish before sending IO ","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851798356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851822944Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Max found gen is 158","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851843212Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851864451Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Next flush: 269","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851884519Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851904697Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"":"downstairs","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851929575Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851953053Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"client":"0","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851976331Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"client":"1","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.851997699Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"client":"2","":"downstairs","session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.852019107Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.852039506Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"3902eb81-e138-4f85-a0d1-89320a414896 is now active with session: 562b8c16-ffda-454c-b47d-4ffe97cb86f0","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.852062024Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.852081282Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838,"session_id":"562b8c16-ffda-454c-b47d-4ffe97cb86f0"} {"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.85210152Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838} {"msg":"Activated sub_volume 3902eb81-e138-4f85-a0d1-89320a414896","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:44.852118899Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 592 len 8192 data: 13 13 02/30 Write block 70 len 24576 data: 15 14 14 13 13 16 03/30 Flush 04/30 Write block 543 len 8192 data: 12 11 05/30 Read block 139 len 8192 06/30 Read block 118 len 12288 07/30 Write block 247 len 20480 data: 12 11 12 14 13 08/30 Read block 55 len 28672 09/30 Read block 180 len 36864 10/30 Write block 519 len 36864 data: 7 8 10 12 10 10 9 10 11 11/30 Write block 227 len 4096 data: 9 12/30 Write block 498 len 24576 data: 9 11 11 10 8 9 13/30 Write block 198 len 12288 data: 18 18 16 14/30 Write block 113 len 16384 data: 12 11 10 9 15/30 Read block 320 len 12288 16/30 Flush 17/30 Flush 18/30 Write block 241 len 24576 data: 10 10 12 9 9 12 19/30 Write block 526 len 16384 data: 11 12 12 13 20/30 Write block 257 len 8192 data: 16 13 21/30 Write block 198 len 12288 data: 19 19 17 22/30 Flush 23/30 Write block 112 len 36864 data: 10 13 12 11 10 13 14 12 12 24/30 Read block 126 len 8192 25/30 Write block 340 len 8192 data: 13 11 26/30 Read block 500 len 28672 27/30 Write block 479 len 20480 data: 7 12 10 14 15 28/30 Read block 512 len 32768 29/30 Read block 56 len 20480 30/30 Write block 305 len 28672 data: 18 17 16 14 11 14 12 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Sent false 1001 Acked Write 6 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 2 Done Done Sent false 1004 Acked Read 2 Done Done Sent false 1005 Acked Read 3 Done Done Sent false 1006 Acked Write 5 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Write 3 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 3 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Read 2 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:44.868892017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1838} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 268 268 268 F F F 1 020-039 154 154 154 265 265 265 F F F 2 040-059 154 154 154 263 263 263 F F F 3 060-079 158 158 156 269 269 267 F F F <--- 4 080-099 157 157 157 268 268 268 F F F 5 100-119 158 158 157 270 270 268 T T F <--- 6 120-139 150 150 150 257 257 257 T T F <--- 7 140-159 157 157 157 268 268 268 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 148 272 272 252 F F F <--- 10 200-219 158 158 157 272 272 268 F F F <--- 11 220-239 158 158 152 270 270 261 F F F <--- 12 240-259 158 158 155 272 272 266 F F F <--- 13 260-279 157 157 157 268 268 268 F F F 14 280-299 154 154 154 265 265 265 F F F 15 300-319 148 148 148 250 250 250 T T F <--- 16 320-339 157 157 157 268 268 268 F F F 17 340-359 154 154 154 264 264 264 T T F <--- 18 360-379 157 157 157 268 268 268 F F F 19 380-399 156 156 156 267 267 267 F F F 20 400-419 154 154 154 265 265 265 F F F 21 420-439 154 154 154 265 265 265 F F F 22 440-459 155 155 155 266 266 266 F F F 23 460-479 152 152 152 261 261 261 T T F <--- 24 480-499 158 158 157 270 270 268 T T F <--- 25 500-519 158 158 150 270 270 258 F F F <--- 26 520-539 158 158 152 272 272 261 F F F <--- 27 540-559 158 158 157 270 270 268 F F F <--- 28 560-579 153 153 153 262 262 262 F F F 29 580-599 158 158 157 269 269 268 F F F <--- Max gen: 158, Max flush: 272 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 157 157 268 268 268 F F F 1 020-039 154 154 154 265 265 265 F F F 2 040-059 154 154 154 263 263 263 F F F 3 060-079 158 158 158 269 269 269 F F F 4 080-099 157 157 157 268 268 268 F F F 5 100-119 159 159 159 273 273 273 F F F 6 120-139 159 159 159 273 273 273 F F F 7 140-159 157 157 157 268 268 268 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 158 158 158 272 272 272 F F F 11 220-239 158 158 158 270 270 270 F F F 12 240-259 158 158 158 272 272 272 F F F 13 260-279 157 157 157 268 268 268 F F F 14 280-299 154 154 154 265 265 265 F F F 15 300-319 159 159 159 273 273 273 F F F 16 320-339 157 157 157 268 268 268 F F F 17 340-359 159 159 159 273 273 273 F F F 18 360-379 157 157 157 268 268 268 F F F 19 380-399 156 156 156 267 267 267 F F F 20 400-419 154 154 154 265 265 265 F F F 21 420-439 154 154 154 265 265 265 F F F 22 440-459 155 155 155 266 266 266 F F F 23 460-479 159 159 159 273 273 273 F F F 24 480-499 159 159 159 273 273 273 F F F 25 500-519 158 158 158 270 270 270 F F F 26 520-539 158 158 158 272 272 272 F F F 27 540-559 158 158 158 270 270 270 F F F 28 560-579 153 153 153 262 262 262 F F F 29 580-599 158 158 158 269 269 269 F F F Max gen: 159, Max flush: 273 Begin loop 80 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.284195732Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:49.327582701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.327994995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.358557812Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.359207405Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.359255451Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.359275749Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.359297907Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846} {"msg":"Crucible fd11bb98-3d2f-411d-8c66-744ea10fcfd8 has session id: bb9a843a-093c-4052-838d-7ddfa1b70dad","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.359821552Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Upstairs opts: Upstairs UUID: fd11bb98-3d2f-411d-8c66-744ea10fcfd8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.359878867Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Crucible stats registered with UUID: fd11bb98-3d2f-411d-8c66-744ea10fcfd8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.359902035Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:49.359918903Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.360100837Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"io task","client":"1","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.360149943Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"io task","client":"0","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.36018128Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.36030352Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"task":"crutest"} {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.360324028Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846} {"msg":"fd11bb98-3d2f-411d-8c66-744ea10fcfd8 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.360391182Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.36041148Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"client":"0","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.360429089Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"client":"1","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.360446447Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"client":"2","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.360467626Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.360493703Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"io task","client":"0","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.36052345Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"io task","client":"1","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"ds_connection connected from Ok(127.0.0.1:60724)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.360577966Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"ds_connection connected from Ok(127.0.0.1:34549)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.360613283Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"io task","client":"0","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"ds_connection connected from Ok(127.0.0.1:64652)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.36064662Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"io task","client":"1","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.361844266Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"client":"2","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.361894081Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"client":"1","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.361936547Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"client":"0","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362431504Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362525126Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"[0]R flush_numbers[0..12]: [268, 265, 263, 269, 268, 273, 273, 268, 268, 272, 272, 270]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.3625915Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"[0]R generation[0..12]: [157, 154, 154, 158, 157, 159, 159, 157, 157, 158, 158, 158]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362615738Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362636266Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"[1]R flush_numbers[0..12]: [268, 265, 263, 269, 268, 273, 273, 268, 268, 272, 272, 270]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362655665Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"[1]R generation[0..12]: [157, 154, 154, 158, 157, 159, 159, 157, 157, 158, 158, 158]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362677653Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362697771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"[2]R flush_numbers[0..12]: [268, 265, 263, 269, 268, 273, 273, 268, 268, 272, 272, 270]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362718019Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"[2]R generation[0..12]: [157, 154, 154, 158, 157, 159, 159, 157, 157, 158, 158, 158]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362737668Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362758196Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362775364Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362792513Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Next flush: 274","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362809221Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.36282581Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"":"downstairs","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362843048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362858667ZWait for a query_work_queue command to finish before sending IO ","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"client":"0","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362882985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"client":"1","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362899743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"client":"2","":"downstairs","session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362916842Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.36293314Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"fd11bb98-3d2f-411d-8c66-744ea10fcfd8 is now active with session: bb9a843a-093c-4052-838d-7ddfa1b70dad","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362954039Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362970017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846,"session_id":"bb9a843a-093c-4052-838d-7ddfa1b70dad"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.362986306Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846} {"msg":"Activated sub_volume fd11bb98-3d2f-411d-8c66-744ea10fcfd8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:49.363000185Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 98 len 24576 data: 18 15 13 15 9 11 02/30 Write block 429 len 12288 data: 14 15 17 03/30 Read block 218 len 20480 04/30 Write block 7 len 16384 data: 14 9 10 14 05/30 Write block 298 len 12288 data: 9 12 13 06/30 Read block 111 len 28672 07/30 Flush 08/30 Write block 539 len 24576 data: 10 13 13 15 13 12 09/30 Read block 155 len 28672 10/30 Read block 530 len 4096 11/30 Read block 228 len 40960 12/30 Write block 401 len 24576 data: 12 11 13 12 13 14 13/30 Write block 443 len 20480 data: 15 16 15 15 17 14/30 Read block 99 len 16384 15/30 Write block 484 len 16384 data: 15 12 13 13 16/30 Write block 417 len 28672 data: 8 10 12 14 12 10 9 17/30 Write block 361 len 40960 data: 16 16 15 16 15 15 14 12 15 10 18/30 Write block 215 len 24576 data: 12 12 14 16 17 17 19/30 Read block 507 len 32768 20/30 Write block 99 len 24576 data: 16 14 16 10 12 9 21/30 Write block 70 len 24576 data: 16 15 15 14 14 17 22/30 Read block 367 len 28672 23/30 Write block 381 len 36864 data: 13 11 11 16 11 10 9 11 9 24/30 Write block 495 len 16384 data: 13 13 11 10 25/30 Read block 515 len 24576 26/30 Write block 591 len 24576 data: 14 14 14 15 14 15 27/30 Write block 374 len 8192 data: 9 11 28/30 Write block 266 len 16384 data: 12 13 12 14 29/30 Write block 14 len 32768 data: 13 13 11 12 13 14 10 11 30/30 Write block 230 len 24576 data: 12 13 15 12 11 10 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Write 4 Sent Done Done false 1004 Acked Write 3 Sent Done Done false 1005 Acked Read 7 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Read 4 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Write 2 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 2 2 31 Done 3 28 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1006)) Some(JobId(1006)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:49.381081859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1846} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 157 160 160 268 274 274 F T T <--- 1 020-039 154 154 154 265 265 265 F T T <--- 2 040-059 154 154 154 263 263 263 F F F 3 060-079 158 158 158 269 269 269 F T T <--- 4 080-099 157 160 160 268 274 274 T T T <--- 5 100-119 159 160 160 273 274 274 T T T <--- 6 120-139 159 159 159 273 273 273 F F F 7 140-159 157 157 157 268 268 268 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 158 158 158 272 272 272 F T T <--- 11 220-239 158 158 158 270 270 270 F T T <--- 12 240-259 158 158 158 272 272 272 F F F 13 260-279 157 157 157 268 268 268 F T T <--- 14 280-299 154 160 160 265 274 274 F F F <--- 15 300-319 159 160 160 273 274 274 F F F <--- 16 320-339 157 157 157 268 268 268 F F F 17 340-359 159 159 159 273 273 273 F F F 18 360-379 157 157 157 268 268 268 F T T <--- 19 380-399 156 156 156 267 267 267 F T T <--- 20 400-419 154 154 154 265 265 265 F T T <--- 21 420-439 154 160 160 265 274 274 T T T <--- 22 440-459 155 155 155 266 266 266 F T T <--- 23 460-479 159 159 159 273 273 273 F F F 24 480-499 159 159 159 273 273 273 F T T <--- 25 500-519 158 158 158 270 270 270 F F F 26 520-539 158 158 158 272 272 272 F T T <--- 27 540-559 158 158 158 270 270 270 F T T <--- 28 560-579 153 153 153 262 262 262 F F F 29 580-599 158 158 158 269 269 269 F T T <--- Max gen: 160, Max flush: 274 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 275 275 275 F F F 1 020-039 161 161 161 275 275 275 F F F 2 040-059 154 154 154 263 263 263 F F F 3 060-079 161 161 161 275 275 275 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 161 161 161 275 275 275 F F F 6 120-139 159 159 159 273 273 273 F F F 7 140-159 157 157 157 268 268 268 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 161 161 161 275 275 275 F F F 11 220-239 161 161 161 275 275 275 F F F 12 240-259 158 158 158 272 272 272 F F F 13 260-279 161 161 161 275 275 275 F F F 14 280-299 160 160 160 274 274 274 F F F 15 300-319 160 160 160 274 274 274 F F F 16 320-339 157 157 157 268 268 268 F F F 17 340-359 159 159 159 273 273 273 F F F 18 360-379 161 161 161 275 275 275 F F F 19 380-399 161 161 161 275 275 275 F F F 20 400-419 161 161 161 275 275 275 F F F 21 420-439 161 161 161 275 275 275 F F F 22 440-459 161 161 161 275 275 275 F F F 23 460-479 159 159 159 273 273 273 F F F 24 480-499 161 161 161 275 275 275 F F F 25 500-519 158 158 158 270 270 270 F F F 26 520-539 161 161 161 275 275 275 F F F 27 540-559 161 161 161 275 275 275 F F F 28 560-579 153 153 153 262 262 262 F F F 29 580-599 161 161 161 275 275 275 F F F Max gen: 161, Max flush: 275 Begin loop 81 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.856887742Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.901603955Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.902317703Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.902365489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.902382047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.902403815Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854} {"msg":"Crucible 20f7a048-288f-443c-8232-413e78811977 has session id: 304a40d1-0a1e-400a-a478-43231a65eca9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.902892893Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Upstairs opts: Upstairs UUID: 20f7a048-288f-443c-8232-413e78811977, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.902943828Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Crucible stats registered with UUID: 20f7a048-288f-443c-8232-413e78811977","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.902978815Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:53.903012442Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903220974Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903262511Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"io task","client":"0","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903282569Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"io task","client":"2","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903427656Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"task":"crutest"} {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903448314Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854} {"msg":"20f7a048-288f-443c-8232-413e78811977 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903504949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903535327Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"client":"0","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903555715Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"client":"1","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903574063Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"client":"2","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903590742Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"io task","client":"2","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.90360843Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"io task","client":"0","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903633718Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"ds_connection connected from Ok(127.0.0.1:43603)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903700832Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"io task","client":"2","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"ds_connection connected from Ok(127.0.0.1:36972)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903741169Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"io task","client":"0","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"ds_connection connected from Ok(127.0.0.1:42686)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.903775976Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.905039176Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"client":"1","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.9051026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"client":"0","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.905618365Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.905716487Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.906246451Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"client":"2","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"[0]R flush_numbers[0..12]: [275, 275, 263, 275, 275, 275, 273, 268, 268, 272, 275, 275]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.906733398Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"[0]R generation[0..12]: [161, 161, 154, 161, 161, 161, 159, 157, 157, 158, 161, 161]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.906764725Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.906785964Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"[1]R flush_numbers[0..12]: [275, 275, 263, 275, 275, 275, 273, 268, 268, 272, 275, 275]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.906813971Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"[1]R generation[0..12]: [161, 161, 154, 161, 161, 161, 159, 157, 157, 158, 161, 161]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.906847248Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.906881825Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"[2]R flush_numbers[0..12]: [275, 275, 263, 275, 275, 275, 273, 268, 268, 272, 275, 275]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.906915532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"[2]R generation[0..12]: [161, 161, 154, 161, 161, 161, 159, 157, 157, 158, 161, 161]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.906950249Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.906972727Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Max found gen is 162","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:22:53.907003285Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.907038602Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Next flush: 276","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.907066219Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.907083607Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"":"downstairs","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.907110615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.907138453Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"client":"0","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.90716607Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"client":"1","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.907184169Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"client":"2","":"downstairs","session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.907212946Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.907242564Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"20f7a048-288f-443c-8232-413e78811977 is now active with session: 304a40d1-0a1e-400a-a478-43231a65eca9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.907270031Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.90729054Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854,"session_id":"304a40d1-0a1e-400a-a478-43231a65eca9"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 crucible","level":30,"time":"2025-12-24T00:22:53.907320737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854} {"msg":"Activated sub_volume 20f7a048-288f-443c-8232-413e78811977","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:53.907344295Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 245 len 24576 data: 10 13 13 12 13 15 02/30 Write block 584 len 8192 data: 8 8 03/30 Write block 307 len 12288 data: 17 15 12 04/30 Read block 176 len 20480 05/30 Flush 06/30 Write block 405 len 4096 data: 14 07/30 Read block 117 len 28672 08/30 Flush 09/30 Write block 141 len 36864 data: 16 15 11 10 11 8 8 10 13 10/30 Write block 329 len 12288 data: 14 12 13 11/30 Write block 512 len 20480 data: 13 12 10 7 8 12/30 Read block 477 len 16384 13/30 Read block 319 len 4096 14/30 Read block 378 len 40960 15/30 Write block 32 len 32768 data: 9 13 9 12 13 13 13 14 16/30 Read block 10 len 16384 17/30 Write block 8 len 4096 data: 10 18/30 Read block 543 len 28672 19/30 Write block 385 len 20480 data: 12 11 10 12 10 20/30 Read block 276 len 16384 21/30 Write block 347 len 28672 data: 13 14 15 14 10 13 14 22/30 Write block 76 len 4096 data: 12 23/30 Read block 158 len 12288 24/30 Write block 282 len 12288 data: 13 13 11 25/30 Write block 339 len 36864 data: 9 14 12 9 8 8 8 10 14 26/30 Write block 360 len 32768 data: 15 17 17 16 17 16 16 15 27/30 Write block 536 len 12288 data: 14 13 12 28/30 Read block 227 len 32768 29/30 Read block 25 len 8192 30/30 Read block 327 len 28672 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Write 3 Done Done Sent false 1003 Acked Read 5 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Read 7 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 9 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Write 5 Done Done Sent false 1011 Acked Read 4 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Read 4 Done Done Sent false 1016 Acked Write 1 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Read 4 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Write 3 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Read 2 Done Done Sent false 1029 Acked Read 7 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 28 28 Done 30 30 2 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) Some(JobId(1007)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:53.92553024Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1854} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 275 275 275 T T F <--- 1 020-039 161 161 161 275 275 275 T T F <--- 2 040-059 154 154 154 263 263 263 F F F 3 060-079 161 161 161 275 275 275 T T F <--- 4 080-099 161 161 161 275 275 275 F F F 5 100-119 161 161 161 275 275 275 F F F 6 120-139 159 159 159 273 273 273 F F F 7 140-159 157 157 157 268 268 268 T T F <--- 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 161 161 161 275 275 275 F F F 11 220-239 161 161 161 275 275 275 F F F 12 240-259 162 162 158 276 276 272 F F T <--- 13 260-279 161 161 161 275 275 275 F F F 14 280-299 160 160 160 274 274 274 T T F <--- 15 300-319 162 162 160 276 276 274 F F F <--- 16 320-339 157 157 157 268 268 268 T T F <--- 17 340-359 159 159 159 273 273 273 T T F <--- 18 360-379 161 161 161 275 275 275 T T F <--- 19 380-399 161 161 161 275 275 275 T T F <--- 20 400-419 162 162 161 277 277 275 F F F <--- 21 420-439 161 161 161 275 275 275 F F F 22 440-459 161 161 161 275 275 275 F F F 23 460-479 159 159 159 273 273 273 F F F 24 480-499 161 161 161 275 275 275 F F F 25 500-519 158 158 158 270 270 270 T T F <--- 26 520-539 161 161 161 275 275 275 T T F <--- 27 540-559 161 161 161 275 275 275 F F F 28 560-579 153 153 153 262 262 262 F F F 29 580-599 162 162 161 276 276 275 F F T <--- Max gen: 162, Max flush: 277 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 163 163 163 278 278 278 F F F 1 020-039 163 163 163 278 278 278 F F F 2 040-059 154 154 154 263 263 263 F F F 3 060-079 163 163 163 278 278 278 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 161 161 161 275 275 275 F F F 6 120-139 159 159 159 273 273 273 F F F 7 140-159 163 163 163 278 278 278 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 161 161 161 275 275 275 F F F 11 220-239 161 161 161 275 275 275 F F F 12 240-259 162 162 162 276 276 276 F F F 13 260-279 161 161 161 275 275 275 F F F 14 280-299 163 163 163 278 278 278 F F F 15 300-319 162 162 162 276 276 276 F F F 16 320-339 163 163 163 278 278 278 F F F 17 340-359 163 163 163 278 278 278 F F F 18 360-379 163 163 163 278 278 278 F F F 19 380-399 163 163 163 278 278 278 F F F 20 400-419 162 162 162 277 277 277 F F F 21 420-439 161 161 161 275 275 275 F F F 22 440-459 161 161 161 275 275 275 F F F 23 460-479 159 159 159 273 273 273 F F F 24 480-499 161 161 161 275 275 275 F F F 25 500-519 163 163 163 278 278 278 F F F 26 520-539 163 163 163 278 278 278 F F F 27 540-559 161 161 161 275 275 275 F F F 28 560-579 153 153 153 262 262 262 F F F 29 580-599 162 162 162 276 276 276 F F F Max gen: 163, Max flush: 278 Begin loop 82 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.350256534Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:58.394103803Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.394505768Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.425688621Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.426365062Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.426423027Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.426443505Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.426471532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862} {"msg":"Crucible 0d0c1ba0-f720-43c5-a663-f9e49b04cee0 has session id: 503b4d99-d429-496b-869a-eae30a5e7508","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.42696427Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Upstairs opts: Upstairs UUID: 0d0c1ba0-f720-43c5-a663-f9e49b04cee0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427035014Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Crucible stats registered with UUID: 0d0c1ba0-f720-43c5-a663-f9e49b04cee0","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427065241Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:58.427086229Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427270863Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427315959Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"io task","client":"1","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427338957Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"io task","client":"2","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427462146Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"task":"crutest"} {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427486694Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862} {"msg":"0d0c1ba0-f720-43c5-a663-f9e49b04cee0 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427546639Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427568307Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"client":"0","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427586665Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"client":"1","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427609833Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"client":"2","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.42764171Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"io task","client":"2","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427676547Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427714034Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"io task","client":"1","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"ds_connection connected from Ok(127.0.0.1:63807)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.42776126Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"ds_connection connected from Ok(127.0.0.1:45674)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427796827Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"io task","client":"2","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"ds_connection connected from Ok(127.0.0.1:49919)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.427832314Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"io task","client":"1","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429047578Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"client":"1","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429080725Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"client":"0","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429166768Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"client":"2","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429788843Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429819301Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"[0]R flush_numbers[0..12]: [278, 278, 263, 278, 275, 275, 273, 278, 268, 272, 275, 275]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429842289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"[0]R generation[0..12]: [163, 163, 154, 163, 161, 161, 159, 163, 157, 158, 161, 161]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429862487Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429882505Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"[1]R flush_numbers[0..12]: [278, 278, 263, 278, 275, 275, 273, 278, 268, 272, 275, 275]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429901524Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"[1]R generation[0..12]: [163, 163, 154, 163, 161, 161, 159, 163, 157, 158, 161, 161]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429920722Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.42994055Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"[2]R flush_numbers[0..12]: [278, 278, 263, 278, 275, 275, 273, 278, 268, 272, 275, 275]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429959819Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"[2]R generation[0..12]: [163, 163, 154, 163, 161, 161, 159, 163, 157, 158, 161, 161]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429978727Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.429998245Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.430023263Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.43005514Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Next flush: 279","v":0,"name":"crucible","level":30,Wait for a query_work_queue command to finish before sending IO "time":"2025-12-24T00:22:58.430087048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.430126134Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"":"downstairs","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.430159621Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.430193948Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"client":"0","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.430228315Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"client":"1","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.430262482Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"client":"2","":"downstairs","session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.43029377Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.430323367Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 0d0c1ba0-f720-43c5-a663-f9e49b04cee0 is now active with session: 503b4d99-d429-496b-869a-eae30a5e7508","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.430356624Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.430381542Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862,"session_id":"503b4d99-d429-496b-869a-eae30a5e7508"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.43039859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862} {"msg":"Activated sub_volume 0d0c1ba0-f720-43c5-a663-f9e49b04cee0","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:22:58.430412859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 204 len 12288 data: 13 14 13 02/30 Read block 125 len 28672 03/30 Read block 96 len 8192 04/30 Read block 515 len 20480 05/30 Write block 235 len 32768 data: 11 9 7 8 10 10 11 11 06/30 Write block 65 len 40960 data: 12 13 15 16 17 17 16 16 15 15 07/30 Read block 486 len 16384 08/30 Read block 308 len 36864 09/30 Write block 110 len 16384 data: 13 13 11 14 10/30 Write block 6 len 32768 data: 14 15 11 11 15 13 13 14 11/30 Read block 43 len 24576 12/30 Read block 196 len 36864 13/30 Read block 570 len 28672 14/30 Read block 218 len 8192 15/30 Read block 442 len 40960 16/30 Write block 592 len 12288 data: 15 15 16 17/30 Read block 394 len 36864 18/30 Read block 171 len 16384 19/30 Write block 438 len 32768 data: 13 14 14 16 15 16 17 16 20/30 Write block 442 len 4096 data: 16 21/30 Write block 100 len 16384 data: 15 17 11 13 22/30 Write block 238 len 20480 data: 9 11 11 12 12 23/30 Read block 515 len 32768 24/30 Read block 589 len 20480 25/30 Write block 277 len 40960 data: 15 12 10 12 14 14 14 12 11 11 26/30 Read block 523 len 16384 27/30 Read block 18 len 4096 28/30 Write block 302 len 36864 data: 16 16 16 19 18 18 16 13 15 29/30 Write block 156 len 8192 data: 12 11 30/30 Write block 357 len 24576 data: 13 13 14 16 18 18 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Write 8 Sent Done Done false 1005 Acked Write 10 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Write 3 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Write 1 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 9 Sent Sent Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 3 3 33 Done 3 27 27 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:22:58.449157116Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1862} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 163 163 163 278 278 278 F T T <--- 1 020-039 163 163 163 278 278 278 F F F 2 040-059 154 154 154 263 263 263 F F F 3 060-079 163 163 163 278 278 278 F T T <--- 4 080-099 161 161 161 275 275 275 F F F 5 100-119 161 161 161 275 275 275 F T T <--- 6 120-139 159 159 159 273 273 273 F F F 7 140-159 163 163 163 278 278 278 F T T <--- 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 161 161 161 275 275 275 T T T 11 220-239 161 161 161 275 275 275 F T T <--- 12 240-259 162 162 162 276 276 276 F T T <--- 13 260-279 161 161 161 275 275 275 F T T <--- 14 280-299 163 163 163 278 278 278 F T T <--- 15 300-319 162 162 162 276 276 276 F T T <--- 16 320-339 163 163 163 278 278 278 F F F 17 340-359 163 163 163 278 278 278 F T T <--- 18 360-379 163 163 163 278 278 278 F T T <--- 19 380-399 163 163 163 278 278 278 F F F 20 400-419 162 162 162 277 277 277 F F F 21 420-439 161 161 161 275 275 275 F T T <--- 22 440-459 161 161 161 275 275 275 F T T <--- 23 460-479 159 159 159 273 273 273 F F F 24 480-499 161 161 161 275 275 275 F F F 25 500-519 163 163 163 278 278 278 F F F 26 520-539 163 163 163 278 278 278 F F F 27 540-559 161 161 161 275 275 275 F F F 28 560-579 153 153 153 262 262 262 F F F 29 580-599 162 162 162 276 276 276 F T T <--- Max gen: 163, Max flush: 278 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 164 164 164 279 279 279 F F F 1 020-039 163 163 163 278 278 278 F F F 2 040-059 154 154 154 263 263 263 F F F 3 060-079 164 164 164 279 279 279 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 164 164 164 279 279 279 F F F 6 120-139 159 159 159 273 273 273 F F F 7 140-159 164 164 164 279 279 279 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 164 164 164 279 279 279 F F F 11 220-239 164 164 164 279 279 279 F F F 12 240-259 164 164 164 279 279 279 F F F 13 260-279 164 164 164 279 279 279 F F F 14 280-299 164 164 164 279 279 279 F F F 15 300-319 164 164 164 279 279 279 F F F 16 320-339 163 163 163 278 278 278 F F F 17 340-359 164 164 164 279 279 279 F F F 18 360-379 164 164 164 279 279 279 F F F 19 380-399 163 163 163 278 278 278 F F F 20 400-419 162 162 162 277 277 277 F F F 21 420-439 164 164 164 279 279 279 F F F 22 440-459 164 164 164 279 279 279 F F F 23 460-479 159 159 159 273 273 273 F F F 24 480-499 161 161 161 275 275 275 F F F 25 500-519 163 163 163 278 278 278 F F F 26 520-539 163 163 163 278 278 278 F F F 27 540-559 161 161 161 275 275 275 F F F 28 560-579 153 153 153 262 262 262 F F F 29 580-599 164 164 164 279 279 279 F F F Max gen: 164, Max flush: 279 Begin loop 83 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.911963421Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.956653347Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.957444558Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.957491564Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.957509032Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.95753108Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870} {"msg":"Crucible 8cb1de95-27f0-493e-9e78-fa3f5a541a4c has session id: c778e518-8148-4b76-8edf-7e3d5a9df1f3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958044046Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Upstairs opts: Upstairs UUID: 8cb1de95-27f0-493e-9e78-fa3f5a541a4c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.95810722Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Crucible stats registered with UUID: 8cb1de95-27f0-493e-9e78-fa3f5a541a4c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958140607Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:02.958159665Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958361738Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958399584Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"2","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958418663Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"0","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958596307Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"task":"crutest"} {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958616556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870} {"msg":"8cb1de95-27f0-493e-9e78-fa3f5a541a4c active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958698429Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958718897Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"client":"0","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958736445Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958755073Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"client":"2","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958783711Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"2","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958854065Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"0","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958892332Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:54888)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958930248Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"0","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:42041)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.958964355Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"2","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:02.959004532Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:02.959046058Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:02.959231862Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.959363Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.959401487Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.960218176Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"client":"2","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.960309328Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"client":"0","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.960821684Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:02.960907546Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.960430126Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"ds_connection connected from Ok(127.0.0.1:63881)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.960571923Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.962478237Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"[0]R flush_numbers[0..12]: [279, 278, 263, 279, 275, 279, 273, 279, 268, 272, 279, 279]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963109842Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"[0]R generation[0..12]: [164, 163, 154, 164, 161, 164, 159, 164, 157, 158, 164, 164]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963139919Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963160648Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"[1]R flush_numbers[0..12]: [279, 278, 263, 279, 275, 279, 273, 279, 268, 272, 279, 279]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963180296Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"[1]R generation[0..12]: [164, 163, 154, 164, 161, 164, 159, 164, 157, 158, 164, 164]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963199554Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963219392Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"[2]R flush_numbers[0..12]: [279, 278, 263, 279, 275, 279, 273, 279, 268, 272, 279, 279]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963238151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"[2]R generation[0..12]: [164, 163, 154, 164, 161, 164, 159, 164, 157, 158, 164, 164]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963257319Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963279337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Max found gen is 165","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963299565Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Generation requested: 166 >= found:165","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963321004Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Next flush: 280","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963341342Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.9633615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963391757Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963414855Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"client":"0","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963436993Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"client":"1","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963458202Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"client":"2","":"downstairs","session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.9634816Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963502018Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"8cb1de95-27f0-493e-9e78-fa3f5a541a4c is now active with session: c778e518-8148-4b76-8edf-7e3d5a9df1f3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963524016Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963542964Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870,"session_id":"c778e518-8148-4b76-8edf-7e3d5a9df1f3"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963563182Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870} {"msg":"Activated sub_volume 8cb1de95-27f0-493e-9e78-fa3f5a541a4c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:12.963580561Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 423 len 20480 data: 10 10 11 13 16 02/30 Write block 119 len 8192 data: 13 13 03/30 Flush 04/30 Write block 340 len 24576 data: 15 13 10 9 9 9 05/30 Write block 348 len 32768 data: 15 16 15 11 14 15 15 16 06/30 Read block 395 len 16384 07/30 Read block 8 len 20480 08/30 Read block 457 len 4096 09/30 Read block 367 len 20480 10/30 Read block 165 len 36864 11/30 Read block 366 len 16384 12/30 Write block 337 len 36864 data: 11 8 10 16 14 11 10 10 10 13/30 Write block 469 len 4096 data: 10 14/30 Read block 115 len 36864 15/30 Write block 564 len 12288 data: 10 9 11 16/30 Write block 29 len 24576 data: 11 11 12 10 14 10 17/30 Write block 248 len 28672 data: 13 14 16 14 13 9 9 18/30 Write block 105 len 40960 data: 13 15 13 13 13 14 14 12 15 13 19/30 Read block 93 len 4096 20/30 Write block 45 len 20480 data: 12 12 11 9 9 21/30 Read block 33 len 32768 22/30 Read block 184 len 32768 23/30 Read block 328 len 32768 24/30 Write block 532 len 16384 data: 13 13 12 11 25/30 Read block 504 len 8192 26/30 Write block 119 len 4096 data: 14 27/30 Write block 292 len 32768 data: 11 10 14 13 11 10 10 13 28/30 Write block 307 len 16384 data: 19 17 14 16 29/30 Read block 537 len 36864 30/30 Write block 365 len 8192 data: 17 17 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 6 Done Sent Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Read 4 Done Sent Done false 1006 Acked Read 5 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Read 5 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Write 10 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Read 9 Done Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 2 30 Done 26 0 25 51 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1002)) Some(JobId(1002)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:12.982316808Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1870} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 164 164 164 279 279 279 F F F 1 020-039 163 163 163 278 278 278 T F T <--- 2 040-059 154 154 154 263 263 263 T F T <--- 3 060-079 164 164 164 279 279 279 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 166 166 166 280 280 280 T F T <--- 6 120-139 166 166 166 280 280 280 F F F 7 140-159 164 164 164 279 279 279 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 164 164 164 279 279 279 F F F 11 220-239 164 164 164 279 279 279 F F F 12 240-259 164 164 164 279 279 279 T F T <--- 13 260-279 164 164 164 279 279 279 F F F 14 280-299 164 164 164 279 279 279 T F T <--- 15 300-319 164 164 164 279 279 279 T F T <--- 16 320-339 163 163 163 278 278 278 T F T <--- 17 340-359 164 164 164 279 279 279 T F T <--- 18 360-379 164 164 164 279 279 279 T F T <--- 19 380-399 163 163 163 278 278 278 F F F 20 400-419 162 162 162 277 277 277 F F F 21 420-439 166 166 166 280 280 280 F F F 22 440-459 164 164 164 279 279 279 F F F 23 460-479 159 159 159 273 273 273 T F T <--- 24 480-499 161 161 161 275 275 275 F F F 25 500-519 163 163 163 278 278 278 F F F 26 520-539 163 163 163 278 278 278 T F T <--- 27 540-559 161 161 161 275 275 275 F F F 28 560-579 153 153 153 262 262 262 T F T <--- 29 580-599 164 164 164 279 279 279 F F F Max gen: 166, Max flush: 280 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 164 164 164 279 279 279 F F F 1 020-039 167 167 167 281 281 281 F F F 2 040-059 167 167 167 281 281 281 F F F 3 060-079 164 164 164 279 279 279 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 167 167 167 281 281 281 F F F 6 120-139 166 166 166 280 280 280 F F F 7 140-159 164 164 164 279 279 279 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 164 164 164 279 279 279 F F F 11 220-239 164 164 164 279 279 279 F F F 12 240-259 167 167 167 281 281 281 F F F 13 260-279 164 164 164 279 279 279 F F F 14 280-299 167 167 167 281 281 281 F F F 15 300-319 167 167 167 281 281 281 F F F 16 320-339 167 167 167 281 281 281 F F F 17 340-359 167 167 167 281 281 281 F F F 18 360-379 167 167 167 281 281 281 F F F 19 380-399 163 163 163 278 278 278 F F F 20 400-419 162 162 162 277 277 277 F F F 21 420-439 166 166 166 280 280 280 F F F 22 440-459 164 164 164 279 279 279 F F F 23 460-479 167 167 167 281 281 281 F F F 24 480-499 161 161 161 275 275 275 F F F 25 500-519 163 163 163 278 278 278 F F F 26 520-539 167 167 167 281 281 281 F F F 27 540-559 161 161 161 275 275 275 F F F 28 560-579 167 167 167 281 281 281 F F F 29 580-599 164 164 164 279 279 279 F F F Max gen: 167, Max flush: 281 Begin loop 84 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.417543537Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.462533917Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.463288381Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.463343416Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.463374433Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.46341125Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878} {"msg":"Crucible c70a561a-9358-456d-9868-f8443b5c8f4f has session id: eeb96dad-bbf8-4239-b813-6e8e60b55a40","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.463959802Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Upstairs opts: Upstairs UUID: c70a561a-9358-456d-9868-f8443b5c8f4f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464028147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Crucible stats registered with UUID: c70a561a-9358-456d-9868-f8443b5c8f4f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464057214Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:17.464075372Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464276755Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464325221Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"io task","client":"0","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464360288Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464479627Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"task":"crutest"} {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464501775Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878} {"msg":"c70a561a-9358-456d-9868-f8443b5c8f4f active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464576289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464601826Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"client":"0","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464635074Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"client":"1","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464665321Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"client":"2","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464693199Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464724296Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"io task","client":"0","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464755753Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"ds_connection connected from Ok(127.0.0.1:38597)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.46479075Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"io task","client":"0","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"ds_connection connected from Ok(127.0.0.1:33589)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464820618Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"ds_connection connected from Ok(127.0.0.1:33793)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.464850395Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.466092017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"client":"0","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.466125324Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"client":"1","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.466683405Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.466765438Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.466794066Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"client":"2","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"[0]R flush_numbers[0..12]: [279, 281, 281, 279, 275, 281, 280, 279, 268, 272, 279, 279]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467349157Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"[0]R generation[0..12]: [164, 167, 167, 164, 161, 167, 166, 164, 157, 158, 164, 164]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467381834Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467402382Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"[1]R flush_numbers[0..12]: [279, 281, 281, 279, 275, 281, 280, 279, 268, 272, 279, 279]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467421441Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"[1]R generation[0..12]: [164, 167, 167, 164, 161, 167, 166, 164, 157, 158, 164, 164]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467440399Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467460057Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"[2]R flush_numbers[0..12]: [279, 281, 281, 279, 275, 281, 280, 279, 268, 272, 279, 279]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467482575Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"[2]R generation[0..12]: [164, 167, 167, 164, 161, 167, 166, 164, 157, 158, 164, 164]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467501724Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:23:17.467521282Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Max found gen is 168","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.46754368Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467561089Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Next flush: 282","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467577187Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467593506Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"":"downstairs","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467610564Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467625963Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"client":"0","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467643012Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"client":"1","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.46765984Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"client":"2","":"downstairs","session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467677289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467693527Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"c70a561a-9358-456d-9868-f8443b5c8f4f is now active with session: eeb96dad-bbf8-4239-b813-6e8e60b55a40","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467711506Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467729694Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878,"session_id":"eeb96dad-bbf8-4239-b813-6e8e60b55a40"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467748562Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878} {"msg":"Activated sub_volume c70a561a-9358-456d-9868-f8443b5c8f4f","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:17.467762951Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 553 len 8192 data: 12 12 02/30 Flush 03/30 Write block 209 len 4096 data: 14 04/30 Write block 145 len 8192 data: 12 9 05/30 Write block 46 len 32768 data: 13 12 10 10 8 8 5 6 06/30 Write block 109 len 32768 data: 14 15 15 13 16 14 12 11 07/30 Read block 394 len 20480 08/30 Write block 371 len 36864 data: 8 8 9 10 12 12 11 11 9 09/30 Read block 176 len 36864 10/30 Read block 212 len 28672 11/30 Write block 59 len 16384 data: 8 7 8 10 12/30 Read block 56 len 32768 13/30 Read block 280 len 20480 14/30 Read block 302 len 16384 15/30 Read block 67 len 40960 16/30 Write block 388 len 8192 data: 13 11 17/30 Read block 64 len 16384 18/30 Write block 239 len 12288 data: 12 12 13 19/30 Read block 400 len 28672 20/30 Write block 278 len 32768 data: 13 11 13 15 15 15 13 12 21/30 Read block 107 len 24576 22/30 Read block 363 len 8192 23/30 Read block 431 len 32768 24/30 Flush 25/30 Write block 229 len 16384 data: 10 13 14 16 26/30 Read block 425 len 16384 27/30 Write block 383 len 32768 data: 12 17 13 12 11 14 12 10 28/30 Write block 355 len 20480 data: 17 13 14 14 15 29/30 Read block 185 len 28672 30/30 Write block 463 len 20480 data: 12 14 12 10 10 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 1 Done Done Done false 1003 Acked Write 2 Done Done Done false 1004 Acked Write 8 Done Done Done false 1005 Acked Write 8 Done Done Done false 1006 Acked Read 5 Done Done Done false 1007 Acked Write 9 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Read 8 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Read 4 Done Done Sent false 1017 Acked Write 3 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 23 25 Done 27 27 5 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:17.486956638Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1878} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 164 164 164 279 279 279 F F F 1 020-039 167 167 167 281 281 281 F F F 2 040-059 168 168 167 283 283 281 F F T <--- 3 060-079 168 168 164 283 283 279 F F F <--- 4 080-099 161 161 161 275 275 275 F F F 5 100-119 168 168 167 283 283 281 F F T <--- 6 120-139 166 166 166 280 280 280 F F F 7 140-159 168 168 164 283 283 279 F F T <--- 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 168 168 164 283 283 279 F F T <--- 11 220-239 168 168 164 283 283 279 T T F <--- 12 240-259 168 168 167 283 283 281 F F F <--- 13 260-279 168 168 164 283 283 279 F F F <--- 14 280-299 168 168 167 283 283 281 F F F <--- 15 300-319 167 167 167 281 281 281 F F F 16 320-339 167 167 167 281 281 281 F F F 17 340-359 167 167 167 281 281 281 T T F <--- 18 360-379 168 168 167 283 283 281 F F F <--- 19 380-399 168 168 163 283 283 278 T T F <--- 20 400-419 162 162 162 277 277 277 F F F 21 420-439 166 166 166 280 280 280 F F F 22 440-459 164 164 164 279 279 279 F F F 23 460-479 167 167 167 281 281 281 T T F <--- 24 480-499 161 161 161 275 275 275 F F F 25 500-519 163 163 163 278 278 278 F F F 26 520-539 167 167 167 281 281 281 F F F 27 540-559 168 168 168 282 282 282 F F F 28 560-579 167 167 167 281 281 281 F F F 29 580-599 164 164 164 279 279 279 F F F Max gen: 168, Max flush: 283 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 164 164 164 279 279 279 F F F 1 020-039 167 167 167 281 281 281 F F F 2 040-059 168 168 168 283 283 283 F F F 3 060-079 168 168 168 283 283 283 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 168 168 168 283 283 283 F F F 6 120-139 166 166 166 280 280 280 F F F 7 140-159 168 168 168 283 283 283 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 168 168 168 283 283 283 F F F 11 220-239 169 169 169 284 284 284 F F F 12 240-259 168 168 168 283 283 283 F F F 13 260-279 168 168 168 283 283 283 F F F 14 280-299 168 168 168 283 283 283 F F F 15 300-319 167 167 167 281 281 281 F F F 16 320-339 167 167 167 281 281 281 F F F 17 340-359 169 169 169 284 284 284 F F F 18 360-379 168 168 168 283 283 283 F F F 19 380-399 169 169 169 284 284 284 F F F 20 400-419 162 162 162 277 277 277 F F F 21 420-439 166 166 166 280 280 280 F F F 22 440-459 164 164 164 279 279 279 F F F 23 460-479 169 169 169 284 284 284 F F F 24 480-499 161 161 161 275 275 275 F F F 25 500-519 163 163 163 278 278 278 F F F 26 520-539 167 167 167 281 281 281 F F F 27 540-559 168 168 168 282 282 282 F F F 28 560-579 167 167 167 281 281 281 F F F 29 580-599 164 164 164 279 279 279 F F F Max gen: 169, Max flush: 284 Begin loop 85 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.896004429Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:21.939547044Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.939967218Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.97104291Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.971789445Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.97184504Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.971864218Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.971891396Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886} {"msg":"Crucible 45077884-123c-4aef-a1af-05f0a78e6397 has session id: 5acf662a-4388-4b7c-acba-9a3a43517c93","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.972498623Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Upstairs opts: Upstairs UUID: 45077884-123c-4aef-a1af-05f0a78e6397, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.972543229Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Crucible stats registered with UUID: 45077884-123c-4aef-a1af-05f0a78e6397","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.972569647Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:21.972588225Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.972788208Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"io task","client":"1","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.972825075Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"io task","client":"0","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.972843823Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973001089Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"task":"crutest"} {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973026557Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886} {"msg":"45077884-123c-4aef-a1af-05f0a78e6397 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973091151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973110939Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"client":"0","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973128288Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"client":"1","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973145336Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"client":"2","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973161995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973188573Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"io task","client":"0","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.97321882Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"io task","client":"1","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"ds_connection connected from Ok(127.0.0.1:65261)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973256427Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"ds_connection connected from Ok(127.0.0.1:44479)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973358368Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"io task","client":"1","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"ds_connection connected from Ok(127.0.0.1:58246)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.973519194Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"io task","client":"0","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.974572412Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"client":"1","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.974606259Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"client":"2","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.974630647Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"client":"0","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975156041Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975251623Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"[0]R flush_numbers[0..12]: [279, 281, 283, 283, 275, 283, 280, 283, 268, 272, 283, 284]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975335406Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"[0]R generation[0..12]: [164, 167, 168, 168, 161, 168, 166, 168, 157, 158, 168, 169]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975361583Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975382632Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"[1]R flush_numbers[0..12]: [279, 281, 283, 283, 275, 283, 280, 283, 268, 272, 283, 284]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.9754017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"[1]R generation[0..12]: [164, 167, 168, 168, 161, 168, 166, 168, 157, 158, 168, 169]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975420558Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975440137Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"[2]R flush_numbers[0..12]: [279, 281, 283, 283, 275, 283, 280, 283, 268, 272, 283, 284]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975459215Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"[2]R generation[0..12]: [164, 167, 168, 168, 161, 168, 166, 168, 157, 158, 168, 169]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975478103Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975497502Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.97551388Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975531019Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Wait for a query_work_queue command to finish before sending IO Next flush: 285","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975556877Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975573775Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"":"downstairs","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975590763Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975606142Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"client":"0","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975623401Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"client":"1","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975640069Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"client":"2","":"downstairs","session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975657258Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975673606Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"45077884-123c-4aef-a1af-05f0a78e6397 is now active with session: 5acf662a-4388-4b7c-acba-9a3a43517c93","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975691525Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975706883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886,"session_id":"5acf662a-4388-4b7c-acba-9a3a43517c93"} {"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975722762Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886} {"msg":"Activated sub_volume 45077884-123c-4aef-a1af-05f0a78e6397","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:21.975736721Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 10 len 28672 data: 16 14 14 15 14 14 12 02/30 Read block 11 len 24576 03/30 Read block 513 len 16384 04/30 Read block 28 len 32768 05/30 Write block 575 len 32768 data: 11 10 9 10 9 9 7 7 06/30 Write block 65 len 36864 data: 13 14 16 17 18 18 17 17 16 07/30 Read block 543 len 12288 08/30 Read block 112 len 36864 09/30 Write block 386 len 40960 data: 13 12 15 13 11 10 8 6 7 10 10/30 Read block 247 len 24576 11/30 Read block 466 len 8192 12/30 Read block 267 len 32768 13/30 Read block 58 len 20480 14/30 Read block 208 len 20480 15/30 Write block 247 len 8192 data: 14 14 16/30 Read block 113 len 32768 17/30 Read block 81 len 36864 18/30 Write block 331 len 16384 data: 14 11 11 10 19/30 Write block 205 len 20480 data: 15 14 13 12 15 20/30 Read block 499 len 4096 21/30 Write block 200 len 16384 data: 18 16 14 15 22/30 Flush 23/30 Write block 464 len 4096 data: 15 24/30 Flush 25/30 Write block 579 len 24576 data: 10 10 8 8 9 9 26/30 Write block 9 len 12288 data: 12 17 15 27/30 Write block 494 len 12288 data: 12 14 14 28/30 Read block 407 len 8192 29/30 Read block 321 len 12288 30/30 Write block 365 len 24576 data: 18 18 16 13 16 11 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Write 8 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Read 3 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Read 2 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 1 1 28 Done 4 29 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:21.995037339Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1886} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 164 170 170 279 285 285 T T T <--- 1 020-039 167 167 167 281 281 281 F F F 2 040-059 168 168 168 283 283 283 F F F 3 060-079 168 170 170 283 285 285 F F F <--- 4 080-099 161 161 161 275 275 275 F F F 5 100-119 168 168 168 283 283 283 F F F 6 120-139 166 166 166 280 280 280 F F F 7 140-159 168 168 168 283 283 283 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 168 170 170 283 285 285 F F F <--- 11 220-239 169 169 169 284 284 284 F F F 12 240-259 168 170 170 283 285 285 F F F <--- 13 260-279 168 168 168 283 283 283 F F F 14 280-299 168 168 168 283 283 283 F F F 15 300-319 167 167 167 281 281 281 F F F 16 320-339 167 170 170 281 285 285 F F F <--- 17 340-359 169 169 169 284 284 284 F F F 18 360-379 168 168 168 283 283 283 F T T <--- 19 380-399 169 170 170 284 285 285 F F F <--- 20 400-419 162 162 162 277 277 277 F F F 21 420-439 166 166 166 280 280 280 F F F 22 440-459 164 164 164 279 279 279 F F F 23 460-479 169 170 170 284 286 286 F F F <--- 24 480-499 161 161 161 275 275 275 F T T <--- 25 500-519 163 163 163 278 278 278 F F F 26 520-539 167 167 167 281 281 281 F F F 27 540-559 168 168 168 282 282 282 F F F 28 560-579 167 170 170 281 285 285 F T T <--- 29 580-599 164 170 170 279 285 285 F T T <--- Max gen: 170, Max flush: 286 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 171 171 287 287 287 F F F 1 020-039 167 167 167 281 281 281 F F F 2 040-059 168 168 168 283 283 283 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 168 168 168 283 283 283 F F F 6 120-139 166 166 166 280 280 280 F F F 7 140-159 168 168 168 283 283 283 F F F 8 160-179 157 157 157 268 268 268 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 170 170 170 285 285 285 F F F 11 220-239 169 169 169 284 284 284 F F F 12 240-259 170 170 170 285 285 285 F F F 13 260-279 168 168 168 283 283 283 F F F 14 280-299 168 168 168 283 283 283 F F F 15 300-319 167 167 167 281 281 281 F F F 16 320-339 170 170 170 285 285 285 F F F 17 340-359 169 169 169 284 284 284 F F F 18 360-379 171 171 171 287 287 287 F F F 19 380-399 170 170 170 285 285 285 F F F 20 400-419 162 162 162 277 277 277 F F F 21 420-439 166 166 166 280 280 280 F F F 22 440-459 164 164 164 279 279 279 F F F 23 460-479 170 170 170 286 286 286 F F F 24 480-499 171 171 171 287 287 287 F F F 25 500-519 163 163 163 278 278 278 F F F 26 520-539 167 167 167 281 281 281 F F F 27 540-559 168 168 168 282 282 282 F F F 28 560-579 171 171 171 287 287 287 F F F 29 580-599 171 171 171 287 287 287 F F F Max gen: 171, Max flush: 287 Begin loop 86 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.424910575Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:26.468311573Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.468747435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.499527933Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.500303495Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.500353641Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.500370659Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.500394137Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894} {"msg":"Crucible ca24cf33-6fa4-4be4-a784-b6c8caf954a5 has session id: 0edbe75f-886e-42e6-bb25-51a9c60d9083","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501003964Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Upstairs opts: Upstairs UUID: ca24cf33-6fa4-4be4-a784-b6c8caf954a5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501044051Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Crucible stats registered with UUID: ca24cf33-6fa4-4be4-a784-b6c8caf954a5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501063089Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:26.501079337Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501326356Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501369462Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"io task","client":"0","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501388681Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"io task","client":"2","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501532498Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"task":"crutest"} {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501552236Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894} {"msg":"ca24cf33-6fa4-4be4-a784-b6c8caf954a5 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.50162069Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501640928Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"client":"0","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501671826Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"client":"1","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501705503Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"client":"2","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.50173999Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"io task","client":"2","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501806644Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"io task","client":"0","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501847761Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"ds_connection connected from Ok(127.0.0.1:65245)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501882567Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"io task","client":"0","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"ds_connection connected from Ok(127.0.0.1:60713)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501913045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"io task","client":"2","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"ds_connection connected from Ok(127.0.0.1:62897)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.501943952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.502976352Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"client":"1","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.50311767Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"client":"0","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503219611Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"client":"2","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503532934Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.5036914Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"[0]R flush_numbers[0..12]: [287, 281, 283, 285, 275, 283, 280, 283, 268, 272, 285, 284]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.50379947Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"[0]R generation[0..12]: [171, 167, 168, 170, 161, 168, 166, 168, 157, 158, 170, 169]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503823878Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503848296Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"[1]R flush_numbers[0..12]: [287, 281, 283, 285, 275, 283, 280, 283, 268, 272, 285, 284]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503868434Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"[1]R generation[0..12]: [171, 167, 168, 170, 161, 168, 166, 168, 157, 158, 170, 169]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503887733Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503907291Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"[2]R flush_numbers[0..12]: [287, 281, 283, 285, 275, 283, 280, 283, 268, 272, 285, 284]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503927139Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"[2]R generation[0..12]: [171, 167, 168, 170, 161, 168, 166, 168, 157, 158, 170, 169]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503946698Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503966506Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {Wait for a query_work_queue command to finish before sending IO "msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.503988734Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Generation requested: 172 >= found:172","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504005822Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Next flush: 288","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504021991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.50403823Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"":"downstairs","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504055268Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504070627Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"client":"0","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504087595Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"client":"1","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504104454Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"client":"2","":"downstairs","session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504123542Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:23:26.504140101Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"ca24cf33-6fa4-4be4-a784-b6c8caf954a5 is now active with session: 0edbe75f-886e-42e6-bb25-51a9c60d9083","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504159199Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504174988Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894,"session_id":"0edbe75f-886e-42e6-bb25-51a9c60d9083"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504191446Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894} {"msg":"Activated sub_volume ca24cf33-6fa4-4be4-a784-b6c8caf954a5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:26.504205435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 552 len 20480 data: 11 13 13 12 12 02/30 Write block 412 len 8192 data: 8 9 03/30 Read block 313 len 32768 04/30 Read block 467 len 16384 05/30 Write block 438 len 20480 data: 14 15 15 17 17 06/30 Read block 318 len 20480 07/30 Read block 371 len 28672 08/30 Write block 124 len 24576 data: 10 9 8 11 10 10 09/30 Read block 246 len 12288 10/30 Read block 169 len 36864 11/30 Read block 250 len 20480 12/30 Write block 165 len 8192 data: 14 16 13/30 Flush 14/30 Write block 303 len 20480 data: 17 17 20 19 20 15/30 Write block 124 len 28672 data: 11 10 9 12 11 11 9 16/30 Read block 533 len 32768 17/30 Read block 477 len 24576 18/30 Write block 492 len 16384 data: 14 12 13 15 19/30 Write block 530 len 4096 data: 18 20/30 Read block 578 len 8192 21/30 Flush 22/30 Write block 31 len 28672 data: 13 11 15 11 13 14 14 23/30 Write block 531 len 16384 data: 16 14 14 13 24/30 Write block 402 len 16384 data: 12 14 13 15 25/30 Read block 590 len 16384 26/30 Read block 378 len 20480 27/30 Write block 265 len 4096 data: 12 28/30 Write block 377 len 36864 data: 12 12 10 13 14 12 13 18 14 29/30 Write block 362 len 4096 data: 19 30/30 Read block 430 len 32768 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Read 4 Sent Done Done false 1004 Acked Write 5 Sent Done Done false 1005 Acked Read 5 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 9 Sent Done Done false 1028 Acked Write 1 Sent Done Done false 1029 Acked Read 8 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 0 27 Done 3 30 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:26.523003207Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1894} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 171 171 287 287 287 F F F 1 020-039 167 167 167 281 281 281 F T T <--- 2 040-059 168 168 168 283 283 283 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 168 168 168 283 283 283 F F F 6 120-139 166 172 172 280 289 289 F F F <--- 7 140-159 168 168 168 283 283 283 F F F 8 160-179 157 172 172 268 288 288 F F F <--- 9 180-199 158 158 158 272 272 272 F F F 10 200-219 170 170 170 285 285 285 F F F 11 220-239 169 169 169 284 284 284 F F F 12 240-259 170 170 170 285 285 285 F F F 13 260-279 168 168 168 283 283 283 F T T <--- 14 280-299 168 168 168 283 283 283 F F F 15 300-319 167 172 172 281 289 289 F F F <--- 16 320-339 170 170 170 285 285 285 F F F 17 340-359 169 169 169 284 284 284 F F F 18 360-379 171 171 171 287 287 287 F T T <--- 19 380-399 170 170 170 285 285 285 F T T <--- 20 400-419 162 172 172 277 288 288 T T T <--- 21 420-439 166 172 172 280 288 288 F F F <--- 22 440-459 164 172 172 279 288 288 F F F <--- 23 460-479 170 170 170 286 286 286 F F F 24 480-499 171 172 172 287 289 289 F F F <--- 25 500-519 163 163 163 278 278 278 F F F 26 520-539 167 172 172 281 289 289 F T T <--- 27 540-559 168 172 172 282 288 288 T F F <--- 28 560-579 171 171 171 287 287 287 F F F 29 580-599 171 171 171 287 287 287 F F F Max gen: 172, Max flush: 289 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 171 171 287 287 287 F F F 1 020-039 173 173 173 290 290 290 F F F 2 040-059 168 168 168 283 283 283 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 168 168 168 283 283 283 F F F 6 120-139 172 172 172 289 289 289 F F F 7 140-159 168 168 168 283 283 283 F F F 8 160-179 172 172 172 288 288 288 F F F 9 180-199 158 158 158 272 272 272 F F F 10 200-219 170 170 170 285 285 285 F F F 11 220-239 169 169 169 284 284 284 F F F 12 240-259 170 170 170 285 285 285 F F F 13 260-279 173 173 173 290 290 290 F F F 14 280-299 168 168 168 283 283 283 F F F 15 300-319 172 172 172 289 289 289 F F F 16 320-339 170 170 170 285 285 285 F F F 17 340-359 169 169 169 284 284 284 F F F 18 360-379 173 173 173 290 290 290 F F F 19 380-399 173 173 173 290 290 290 F F F 20 400-419 173 173 173 290 290 290 F F F 21 420-439 172 172 172 288 288 288 F F F 22 440-459 172 172 172 288 288 288 F F F 23 460-479 170 170 170 286 286 286 F F F 24 480-499 172 172 172 289 289 289 F F F 25 500-519 163 163 163 278 278 278 F F F 26 520-539 173 173 173 290 290 290 F F F 27 540-559 172 172 172 288 288 288 F F F 28 560-579 171 171 171 287 287 287 F F F 29 580-599 171 171 171 287 287 287 F F F Max gen: 173, Max flush: 290 Begin loop 87 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:30.962129767Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.00718546Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.007948794Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.00799485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.008011218Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.008033617Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902} {"msg":"Crucible 0a423e1f-eb15-4bae-aae7-33e25a86c9a9 has session id: ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.008534113Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Upstairs opts: Upstairs UUID: 0a423e1f-eb15-4bae-aae7-33e25a86c9a9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.008577289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Crucible stats registered with UUID: 0a423e1f-eb15-4bae-aae7-33e25a86c9a9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.008601227Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:31.008618395Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.008859105Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"io task","client":"0","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.00890443Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.008924599Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"io task","client":"2","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009097474Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"task":"crutest"} {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009124991Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902} {"msg":"0a423e1f-eb15-4bae-aae7-33e25a86c9a9 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009200515Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009221363Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"client":"0","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009239361Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"client":"1","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.00925652Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"client":"2","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009281468Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"io task","client":"2","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009315125Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"io task","client":"0","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009347512Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"ds_connection connected from Ok(127.0.0.1:59057)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009467621Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"io task","client":"2","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"ds_connection connected from Ok(127.0.0.1:61461)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009515907Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"io task","client":"0","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"ds_connection connected from Ok(127.0.0.1:44786)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.009563033Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.010703564Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"client":"1","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.010736701Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"client":"0","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011023246Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"client":"2","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011343208Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011419961Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"[0]R flush_numbers[0..12]: [287, 290, 283, 285, 275, 283, 289, 283, 288, 272, 285, 284]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011570168Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"[0]R generation[0..12]: [171, 173, 168, 170, 161, 168, 172, 168, 172, 158, 170, 169]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011603095Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011628913Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"[1]R flush_numbers[0..12]: [287, 290, 283, 285, 275, 283, 289, 283, 288, 272, 285, 284]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011652921Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"[1]R generation[0..12]: [171, 173, 168, 170, 161, 168, 172, 168, 172, 158, 170, 169]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011676459Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011701017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"[2]R flush_numbers[0..12]: [287, 290, 283, 285, 275, 283, 289, 283, 288, 272, 285, 284]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011729644Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg"Wait for a query_work_queue command to finish before sending IO :"[2]R generation[0..12]: [171, 173, 168, 170, 161, 168, 172, 168, 172, 158, 170, 169]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.01177736Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011827546Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011866992Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.01189986Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Next flush: 291","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011919328Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.011942566Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"":"downstairs","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"No downstairs reconciliation required","v":0,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 name":"crucible","level":30,"time":"2025-12-24T00:23:31.011977353Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.01200597Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"client":"0","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.012024859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"client":"1","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.012042087Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"client":"2","":"downstairs","session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.012059486Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.012075874Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"0a423e1f-eb15-4bae-aae7-33e25a86c9a9 is now active with session: ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.012093983Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.012109421Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902,"session_id":"ab4985f5-56ed-4e1e-a78c-0e4a0c940ee6"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.01212559Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902} {"msg":"Activated sub_volume 0a423e1f-eb15-4bae-aae7-33e25a86c9a9","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:31.012139739Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 587 len 16384 data: 11 10 11 13 02/30 Write block 55 len 4096 data: 4 03/30 Write block 387 len 12288 data: 13 16 14 04/30 Read block 157 len 36864 05/30 Read block 275 len 20480 06/30 Write block 140 len 40960 data: 14 17 16 12 11 13 10 9 11 14 07/30 Write block 422 len 8192 data: 11 11 08/30 Write block 231 len 8192 data: 15 17 09/30 Write block 322 len 8192 data: 13 13 10/30 Read block 295 len 28672 11/30 Write block 494 len 12288 data: 14 16 15 12/30 Read block 153 len 4096 13/30 Write block 142 len 16384 data: 17 13 12 14 14/30 Read block 397 len 28672 15/30 Write block 421 len 24576 data: 13 12 12 11 12 14 16/30 Write block 485 len 20480 data: 13 14 14 9 8 17/30 Write block 138 len 4096 data: 14 18/30 Write block 6 len 8192 data: 15 16 19/30 Write block 588 len 36864 data: 11 12 14 15 16 16 17 15 16 20/30 Write block 513 len 8192 data: 13 11 21/30 Write block 5 len 36864 data: 11 16 17 12 13 18 16 15 16 22/30 Write block 188 len 16384 data: 12 14 12 12 23/30 Write block 344 len 8192 data: 11 11 24/30 Read block 476 len 28672 25/30 Flush 26/30 Write block 1 len 12288 data: 4 5 6 27/30 Write block 408 len 8192 data: 13 12 28/30 Read block 573 len 32768 29/30 Read block 588 len 12288 30/30 Read block 271 len 8192 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Write 3 Done Done Done false 1003 Acked Read 9 Done Done Sent false 1004 Acked Read 5 Done Done Sent false 1005 Acked Write 10 Done Done Sent false 1006 Acked Write 2 Done Done Sent false 1007 Acked Write 2 Done Done Sent false 1008 Acked Write 2 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Write 3 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Write 4 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Write 1 Done Done Sent false 1017 Acked Write 2 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 27 27 Done 30 30 3 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:31.02715831Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1902} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 174 174 171 291 291 287 T T F <--- 1 020-039 173 173 173 290 290 290 F F F 2 040-059 174 174 168 291 291 283 F F T <--- 3 060-079 170 170 170 285 285 285 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 168 168 168 283 283 283 F F F 6 120-139 174 174 172 291 291 289 F F F <--- 7 140-159 174 174 168 291 291 283 F F F <--- 8 160-179 172 172 172 288 288 288 F F F 9 180-199 174 174 158 291 291 272 F F F <--- 10 200-219 170 170 170 285 285 285 F F F 11 220-239 174 174 169 291 291 284 F F F <--- 12 240-259 170 170 170 285 285 285 F F F 13 260-279 173 173 173 290 290 290 F F F 14 280-299 168 168 168 283 283 283 F F F 15 300-319 172 172 172 289 289 289 F F F 16 320-339 174 174 170 291 291 285 F F F <--- 17 340-359 174 174 169 291 291 284 F F F <--- 18 360-379 173 173 173 290 290 290 F F F 19 380-399 174 174 173 291 291 290 F F T <--- 20 400-419 173 173 173 290 290 290 T T F <--- 21 420-439 174 174 172 291 291 288 F F F <--- 22 440-459 172 172 172 288 288 288 F F F 23 460-479 170 170 170 286 286 286 F F F 24 480-499 174 174 172 291 291 289 F F F <--- 25 500-519 174 174 163 291 291 278 F F F <--- 26 520-539 173 173 173 290 290 290 F F F 27 540-559 172 172 172 288 288 288 F F F 28 560-579 171 171 171 287 287 287 F F F 29 580-599 174 174 171 291 291 287 F F T <--- Max gen: 174, Max flush: 291 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 292 292 292 F F F 1 020-039 173 173 173 290 290 290 F F F 2 040-059 174 174 174 291 291 291 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 161 161 161 275 275 275 F F F 5 100-119 168 168 168 283 283 283 F F F 6 120-139 174 174 174 291 291 291 F F F 7 140-159 174 174 174 291 291 291 F F F 8 160-179 172 172 172 288 288 288 F F F 9 180-199 174 174 174 291 291 291 F F F 10 200-219 170 170 170 285 285 285 F F F 11 220-239 174 174 174 291 291 291 F F F 12 240-259 170 170 170 285 285 285 F F F 13 260-279 173 173 173 290 290 290 F F F 14 280-299 168 168 168 283 283 283 F F F 15 300-319 172 172 172 289 289 289 F F F 16 320-339 174 174 174 291 291 291 F F F 17 340-359 174 174 174 291 291 291 F F F 18 360-379 173 173 173 290 290 290 F F F 19 380-399 174 174 174 291 291 291 F F F 20 400-419 175 175 175 292 292 292 F F F 21 420-439 174 174 174 291 291 291 F F F 22 440-459 172 172 172 288 288 288 F F F 23 460-479 170 170 170 286 286 286 F F F 24 480-499 174 174 174 291 291 291 F F F 25 500-519 174 174 174 291 291 291 F F F 26 520-539 173 173 173 290 290 290 F F F 27 540-559 172 172 172 288 288 288 F F F 28 560-579 171 171 171 287 287 287 F F F 29 580-599 174 174 174 291 291 291 F F F Max gen: 175, Max flush: 292 Begin loop 88 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.443733985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:35.487193098Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.487585534Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.518314496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.518946971Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.518992847Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.519008476Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.519030334Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910} {"msg":"Crucible a406d935-7eea-4d72-894e-efa28bb23000 has session id: 990ddf95-2c9f-4215-aeca-0d10119c3df4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.519612353Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Upstairs opts: Upstairs UUID: a406d935-7eea-4d72-894e-efa28bb23000, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.519662928Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Crucible stats registered with UUID: a406d935-7eea-4d72-894e-efa28bb23000","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.519684627Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:35.519702045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.5198714Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"io task","client":"2","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.519909997Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"io task","client":"0","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.519929555Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.520119979Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"task":"crutest"} {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.520146926Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910} {"msg":"a406d935-7eea-4d72-894e-efa28bb23000 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.520209911Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.520230349Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"client":"0","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.520248017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"client":"1","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.520265066Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"client":"2","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.520281965Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"io task","client":"2","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.52033949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"io task","client":"0","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.520361878Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"ds_connection connected from Ok(127.0.0.1:33591)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.520383236Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"io task","client":"2","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"ds_connection connected from Ok(127.0.0.1:46983)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.520410793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"ds_connection connected from Ok(127.0.0.1:34845)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.5204437Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"io task","client":"0","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.521743277Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"client":"1","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.521779444Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"client":"0","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.521800172Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"client":"2","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522342115Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522375952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"[0]R flush_numbers[0..12]: [292, 290, 291, 285, 275, 283, 291, 291, 288, 291, 285, 291]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.52240216Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"[0]R generation[0..12]: [175, 173, 174, 170, 161, 168, 174, 174, 172, 174, 170, 174]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522427488Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522448246Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"[1]R flush_numbers[0..12]: [292, 290, 291, 285, 275, 283, 291, 291, 288, 291, 285, 291]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522467414Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"[1]R generation[0..12]: [175, 173, 174, 170, 161, 168, 174, 174, 172, 174, 170, 174]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522486402Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522506471Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"[2]R flush_numbers[0..12]: [292, 290, 291, 285, 275, 283, 291, 291, 288, 291, 285, 291]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522525629Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"[2]R generation[0..12]: [175, 173, 174, 170, 161, 168, 174, 174, 172, 174, 170, 174]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522545397Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522565036Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Max found gen is 176","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522581464Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522598533Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Next flush: 293","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522620901Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522637599Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"":"downstairs","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522657178Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522672866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"client":"0","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522689955Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"client":"1","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522706933Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"client":"2","":"downstairs","session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522724082Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.52274042Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"a406d935-7eea-4d72-894e-efa28bb23000 is now active with session: 990ddf95-2c9f-4215-aeca-0d10119c3df4","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522758519Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522773868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910,"session_id":"990ddf95-2c9f-4215-aeca-0d10119c3df4"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522789896Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910} {"msg":"Activated sub_volume a406d935-7eea-4d72-894e-efa28bb23000","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:35.522804125Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 160 len 8192 data: 16 20 02/30 Read block 346 len 8192 03/30 Write block 87 len 8192 data: 8 7 04/30 Write block 239 len 32768 data: 13 13 14 13 13 10 11 14 05/30 Read block 240 len 24576 06/30 Read block 555 len 36864 07/30 Read block 520 len 8192 08/30 Write block 299 len 36864 data: 14 14 14 17 18 18 21 20 21 09/30 Write block 299 len 16384 data: 15 15 15 18 10/30 Write block 287 len 12288 data: 11 11 9 11/30 Read block 513 len 20480 12/30 Write block 249 len 20480 data: 15 17 15 14 10 13/30 Flush 14/30 Write block 154 len 36864 data: 12 11 13 12 10 13 17 21 17 15/30 Flush 16/30 Write block 460 len 32768 data: 10 10 11 13 16 13 11 11 17/30 Write block 149 len 4096 data: 15 18/30 Write block 212 len 16384 data: 13 12 13 13 19/30 Write block 404 len 32768 data: 14 16 15 14 14 13 11 12 20/30 Write block 128 len 16384 data: 12 12 10 13 21/30 Read block 350 len 28672 22/30 Write block 572 len 8192 data: 11 12 23/30 Write block 123 len 4096 data: 12 24/30 Write block 44 len 12288 data: 11 13 14 25/30 Read block 409 len 4096 26/30 Write block 226 len 4096 data: 8 27/30 Write block 5 len 36864 data: 12 17 18 13 14 19 17 16 17 28/30 Read block 377 len 24576 29/30 Read block 34 len 28672 30/30 Read block 98 len 32768 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Sent Done Done false 1001 Acked Read 2 Sent Done Done false 1002 Acked Write 2 Sent Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 8 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:35.539991397Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1910} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 175 175 175 292 292 292 F T T <--- 1 020-039 173 173 173 290 290 290 F F F 2 040-059 174 174 174 291 291 291 F T T <--- 3 060-079 170 170 170 285 285 285 F F F 4 080-099 161 176 176 275 293 293 F F F <--- 5 100-119 168 168 168 283 283 283 F F F 6 120-139 174 174 174 291 291 291 F T T <--- 7 140-159 174 176 176 291 294 294 F T T <--- 8 160-179 172 176 176 288 294 294 F F F <--- 9 180-199 174 174 174 291 291 291 F F F 10 200-219 170 170 170 285 285 285 F T T <--- 11 220-239 174 176 176 291 293 293 F T T <--- 12 240-259 170 176 176 285 293 293 F F F <--- 13 260-279 173 173 173 290 290 290 F F F 14 280-299 168 176 176 283 293 293 F F F <--- 15 300-319 172 176 176 289 293 293 F F F <--- 16 320-339 174 174 174 291 291 291 F F F 17 340-359 174 174 174 291 291 291 F F F 18 360-379 173 173 173 290 290 290 F F F 19 380-399 174 174 174 291 291 291 F F F 20 400-419 175 175 175 292 292 292 F T T <--- 21 420-439 174 174 174 291 291 291 F F F 22 440-459 172 172 172 288 288 288 F F F 23 460-479 170 170 170 286 286 286 F T T <--- 24 480-499 174 174 174 291 291 291 F F F 25 500-519 174 174 174 291 291 291 F F F 26 520-539 173 173 173 290 290 290 F F F 27 540-559 172 172 172 288 288 288 F F F 28 560-579 171 171 171 287 287 287 F T T <--- 29 580-599 174 174 174 291 291 291 F F F Max gen: 176, Max flush: 294 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 295 295 295 F F F 1 020-039 173 173 173 290 290 290 F F F 2 040-059 177 177 177 295 295 295 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 168 168 168 283 283 283 F F F 6 120-139 177 177 177 295 295 295 F F F 7 140-159 177 177 177 295 295 295 F F F 8 160-179 176 176 176 294 294 294 F F F 9 180-199 174 174 174 291 291 291 F F F 10 200-219 177 177 177 295 295 295 F F F 11 220-239 177 177 177 295 295 295 F F F 12 240-259 176 176 176 293 293 293 F F F 13 260-279 173 173 173 290 290 290 F F F 14 280-299 176 176 176 293 293 293 F F F 15 300-319 176 176 176 293 293 293 F F F 16 320-339 174 174 174 291 291 291 F F F 17 340-359 174 174 174 291 291 291 F F F 18 360-379 173 173 173 290 290 290 F F F 19 380-399 174 174 174 291 291 291 F F F 20 400-419 177 177 177 295 295 295 F F F 21 420-439 174 174 174 291 291 291 F F F 22 440-459 172 172 172 288 288 288 F F F 23 460-479 177 177 177 295 295 295 F F F 24 480-499 174 174 174 291 291 291 F F F 25 500-519 174 174 174 291 291 291 F F F 26 520-539 173 173 173 290 290 290 F F F 27 540-559 172 172 172 288 288 288 F F F 28 560-579 177 177 177 295 295 295 F F F 29 580-599 174 174 174 291 291 291 F F F Max gen: 177, Max flush: 295 Begin loop 89 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:39.982808974Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:40.026443491Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.026863975Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.057773771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.058508117Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.058565912Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.05859021Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.058618198Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918} {"msg":"Crucible 23cddb88-14a7-4b2a-9d62-e9b75afa4785 has session id: 56ff79c3-734b-41b3-93a0-1deaf904bdb5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059120834Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Upstairs opts: Upstairs UUID: 23cddb88-14a7-4b2a-9d62-e9b75afa4785, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059178919Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Crucible stats registered with UUID: 23cddb88-14a7-4b2a-9d62-e9b75afa4785","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059211316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:40.059238464Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059423717Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"io task","client":"0","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059464844Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"io task","client":"1","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059497311Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059676985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"task":"crutest"} {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059697294Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918} {"msg":"23cddb88-14a7-4b2a-9d62-e9b75afa4785 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059760618Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059780986Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"client":"0","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059798985Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"client":"1","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059816083Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"client":"2","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059840111Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059870258Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"io task","client":"0","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059902416Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"io task","client":"1","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"ds_connection connected from Ok(127.0.0.1:51425)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.059940583Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"ds_connection connected from Ok(127.0.0.1:43092)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.05997121Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"io task","client":"0","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"ds_connection connected from Ok(127.0.0.1:45157)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.060003847Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"io task","client":"1","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.061315872Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"client":"1","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.061364418Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"client":"2","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.061404715Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"client":"0","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.061898772Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.061928459Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"[0]R flush_numbers[0..12]: [295, 290, 295, 285, 293, 283, 295, 295, 294, 291, 295, 295]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.061947548Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"[0]R generation[0..12]: [177, 173, 177, 170, 176, 168, 177, 177, 176, 174, 177, 177]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.061967646Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.061987634Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"[1]R flush_numbers[0..12]: [295, 290, 295, 285, 293, 283, 295, 295, 294, 291, 295, 295]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062010082Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"[1]R generation[0..12]: [177, 173, 177, 170, 176, 168, 177, 177, 176, 174, 177, 177]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.06202973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062049669Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"[2]R flush_numbers[0..12]: [295, 290, 295, 285, 293, 283, 295, 295, 294, 291, 295, 295]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062068827Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"[2]R generation[0..12]: [177, 173, 177, 170, 176, 168, 177, 177, 176, 174, 177, 177]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062088165Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062107734Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062124202Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.06214884Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {Wait for a query_work_queue command to finish before sending IO "msg":"Next flush: 296","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062182497Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062200886Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"":"downstairs","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062218484Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062243602Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"client":"0","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062268569Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"client":"1","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062286418Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"client":"2","":"downstairs","session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062304197Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062335094Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"23cddb88-14a7-4b2a-9d62-e9b75afa4785 is now active with session: 56ff79c3-734b-41b3-93a0-1deaf904bdb5","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062357802Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.0623738Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918,"session_id":"56ff79c3-734b-41b3-93a0-1deaf904bdb5"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:40.062397778Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918} {"msg":"Activated sub_volume 23cddb88-14a7-4b2a-9d62-e9b75afa4785","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:23:40.062420346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 201 len 24576 data: 17 15 16 14 16 15 02/30 Write block 592 len 28672 data: 17 17 18 16 17 12 11 03/30 Write block 584 len 8192 data: 10 9 04/30 Write block 242 len 20480 data: 14 14 11 12 15 05/30 Read block 73 len 16384 06/30 Write block 498 len 12288 data: 11 12 12 07/30 Read block 117 len 4096 08/30 Read block 37 len 8192 09/30 Flush 10/30 Write block 273 len 40960 data: 15 15 14 17 16 14 12 14 16 16 11/30 Write block 492 len 12288 data: 15 13 15 12/30 Read block 564 len 12288 13/30 Write block 196 len 12288 data: 17 17 20 14/30 Flush 15/30 Write block 557 len 4096 data: 7 16/30 Write block 481 len 24576 data: 11 15 16 16 14 15 17/30 Flush 18/30 Write block 344 len 20480 data: 12 12 11 15 16 19/30 Read block 292 len 32768 20/30 Write block 185 len 36864 data: 10 10 14 13 15 13 13 14 15 21/30 Read block 67 len 4096 22/30 Flush 23/30 Write block 115 len 4096 data: 13 24/30 Read block 191 len 40960 25/30 Write block 372 len 28672 data: 9 10 11 13 13 13 13 26/30 Write block 443 len 24576 data: 17 18 17 16 18 16 27/30 Read block 203 len 36864 28/30 Write block 376 len 28672 data: 14 14 14 11 14 15 13 29/30 Write block 437 len 16384 data: 15 15 16 16 30/30 Read block 384 len 36864 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Read 4 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 10 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Write 4 Sent Done Done false 1029 Acked Read 9 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 1 0 27 Done 4 29 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:40.080756289Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1918} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 295 295 295 F F F 1 020-039 173 173 173 290 290 290 F F F 2 040-059 177 177 177 295 295 295 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 168 168 168 283 283 283 F T T <--- 6 120-139 177 177 177 295 295 295 F F F 7 140-159 177 177 177 295 295 295 F F F 8 160-179 176 176 176 294 294 294 F F F 9 180-199 174 178 178 291 299 299 F F F <--- 10 200-219 177 178 178 295 296 296 T F F <--- 11 220-239 177 177 177 295 295 295 F F F 12 240-259 176 178 178 293 296 296 T F F <--- 13 260-279 173 178 178 290 297 297 F F F <--- 14 280-299 176 178 178 293 297 297 F F F <--- 15 300-319 176 176 176 293 293 293 F F F 16 320-339 174 174 174 291 291 291 F F F 17 340-359 174 178 178 291 299 299 F F F <--- 18 360-379 173 173 173 290 290 290 F T T <--- 19 380-399 174 174 174 291 291 291 F T T <--- 20 400-419 177 177 177 295 295 295 F F F 21 420-439 174 174 174 291 291 291 F T T <--- 22 440-459 172 172 172 288 288 288 F T T <--- 23 460-479 177 177 177 295 295 295 F F F 24 480-499 174 178 178 291 298 298 F F F <--- 25 500-519 174 178 178 291 296 296 F F F <--- 26 520-539 173 173 173 290 290 290 F F F 27 540-559 172 178 178 288 298 298 F F F <--- 28 560-579 177 177 177 295 295 295 F F F 29 580-599 174 178 178 291 296 296 T F F <--- Max gen: 178, Max flush: 299 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 295 295 295 F F F 1 020-039 173 173 173 290 290 290 F F F 2 040-059 177 177 177 295 295 295 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 179 179 179 300 300 300 F F F 6 120-139 177 177 177 295 295 295 F F F 7 140-159 177 177 177 295 295 295 F F F 8 160-179 176 176 176 294 294 294 F F F 9 180-199 178 178 178 299 299 299 F F F 10 200-219 178 178 178 296 296 296 F F F 11 220-239 177 177 177 295 295 295 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 178 178 178 297 297 297 F F F 14 280-299 178 178 178 297 297 297 F F F 15 300-319 176 176 176 293 293 293 F F F 16 320-339 174 174 174 291 291 291 F F F 17 340-359 178 178 178 299 299 299 F F F 18 360-379 179 179 179 300 300 300 F F F 19 380-399 179 179 179 300 300 300 F F F 20 400-419 177 177 177 295 295 295 F F F 21 420-439 179 179 179 300 300 300 F F F 22 440-459 179 179 179 300 300 300 F F F 23 460-479 177 177 177 295 295 295 F F F 24 480-499 178 178 178 298 298 298 F F F 25 500-519 178 178 178 296 296 296 F F F 26 520-539 173 173 173 290 290 290 F F F 27 540-559 178 178 178 298 298 298 F F F 28 560-579 177 177 177 295 295 295 F F F 29 580-599 178 178 178 296 296 296 F F F Max gen: 179, Max flush: 300 Begin loop 90 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.530813327Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.575704155Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.576441331Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.576499906Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.576520444Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.576547541Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926} {"msg":"Crucible 383ac765-3ae7-4d62-94e7-0b287b5100f1 has session id: 4b1efdf0-8c29-4296-b900-369a988c7afa","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577058867Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Upstairs opts: Upstairs UUID: 383ac765-3ae7-4d62-94e7-0b287b5100f1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577115102Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Crucible stats registered with UUID: 383ac765-3ae7-4d62-94e7-0b287b5100f1","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577146809Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:44.577179946Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577376359Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577418665Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"io task","client":"2","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577439394Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"io task","client":"1","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577638776Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"task":"crutest"} {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577658895Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926} {"msg":"383ac765-3ae7-4d62-94e7-0b287b5100f1 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577755486Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577790343Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"client":"0","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.57782055Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"client":"1","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577850868Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"client":"2","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577882355Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"io task","client":"2","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577915812Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.57794585Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"io task","client":"1","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"ds_connection connected from Ok(127.0.0.1:49138)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.577978047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"io task","client":"2","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"ds_connection connected from Ok(127.0.0.1:46226)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.578011834Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"ds_connection connected from Ok(127.0.0.1:57562)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.578150132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"io task","client":"1","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.579178502Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"client":"0","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.579303961Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"client":"1","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.579756202Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.579855113Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580088423Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"client":"2","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"[0]R flush_numbers[0..12]: [295, 290, 295, 285, 293, 300, 295, 295, 294, 299, 296, 295]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580593189Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"[0]R generation[0..12]: [177, 173, 177, 170, 176, 179, 177, 177, 176, 178, 178, 177]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580624376Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580645824Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"[1]R flush_numbers[0..12]: [295, 290, 295, 285, 293, 300, 295, 295, 294, 299, 296, 295]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580673092Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"[1]R generation[0..12]: [177, 173, 177, 170, 176, 179, 177, 177, 176, 178, 178, 177]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580706709Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580743106Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"[2]R flush_numbers[0..12]: [295, 290, 295, 285, 293, 300, 295, 295, 294, 299, 296, 295]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580777623Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"[2]R generation[0..12]: [177, 173, 177, 170, 176, 179, 177, 177, 176, 178, 178, 177]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580804861Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580833188Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Max found gen is 180","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580872045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Generation requested: 180 >= found:180","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580902472Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Next flush: 301","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.58092172Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580938499Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"":"downstairs","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580964127Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.580992114Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"client":"0","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.581016602Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"client":"1","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.58103431Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"client":"2","":"downstairs","session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.581063458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.581092485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"383ac765-3ae7-4d62-94e7-0b287b5100f1 is now active with session: 4b1efdf0-8c29-4296-b900-369a988c7afa","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.581119903Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.581136461Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926,"session_id":"4b1efdf0-8c29-4296-b900-369a988c7afa"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:44.581160299Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926} {"msg":"Activated sub_volume 383ac765-3ae7-4d62-94e7-0b287b5100f1","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-12-24T00:23:44.581188077Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 566 len 40960 data: 12 11 12 15 14 11 12 13 12 12 02/30 Read block 467 len 32768 03/30 Read block 133 len 16384 04/30 Read block 83 len 36864 05/30 Write block 125 len 24576 data: 11 10 13 13 13 11 06/30 Read block 31 len 28672 07/30 Write block 415 len 32768 data: 12 11 9 11 13 15 14 13 08/30 Read block 180 len 24576 09/30 Read block 417 len 8192 10/30 Read block 502 len 28672 11/30 Read block 281 len 28672 12/30 Read block 208 len 16384 13/30 Read block 560 len 40960 14/30 Write block 204 len 8192 data: 15 17 15/30 Read block 314 len 8192 16/30 Read block 208 len 8192 17/30 Flush 18/30 Write block 552 len 16384 data: 12 14 14 13 19/30 Read block 397 len 20480 20/30 Write block 195 len 4096 data: 14 21/30 Write block 170 len 40960 data: 14 14 13 11 11 10 13 12 11 10 22/30 Read block 94 len 20480 23/30 Read block 125 len 4096 24/30 Read block 114 len 20480 25/30 Read block 7 len 28672 26/30 Write block 264 len 20480 data: 11 13 13 14 13 27/30 Read block 480 len 36864 28/30 Write block 341 len 20480 data: 15 12 11 13 13 29/30 Write block 1 len 4096 data: 5 30/30 Write block 486 len 4096 data: 16 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Write 6 Done Done Done false 1005 Acked Read 7 Done Done Done false 1006 Acked Write 8 Done Done Done false 1007 Acked Read 6 Done Done Done false 1008 Acked Read 2 Done Done Done false 1009 Acked Read 7 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Read 4 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Read 5 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Read 7 Done Done Sent false 1025 Acked Write 5 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Write 5 Sent Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 21 27 Done 27 27 9 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:44.600969743Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1926} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 295 295 295 T T F <--- 1 020-039 173 173 173 290 290 290 F F F 2 040-059 177 177 177 295 295 295 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 179 179 179 300 300 300 F F F 6 120-139 180 180 177 301 301 295 F F T <--- 7 140-159 177 177 177 295 295 295 F F F 8 160-179 176 176 176 294 294 294 T T F <--- 9 180-199 178 178 178 299 299 299 T T F <--- 10 200-219 180 180 178 301 301 296 F F F <--- 11 220-239 177 177 177 295 295 295 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 178 178 178 297 297 297 T T F <--- 14 280-299 178 178 178 297 297 297 F F F 15 300-319 176 176 176 293 293 293 F F F 16 320-339 174 174 174 291 291 291 F F F 17 340-359 178 178 178 299 299 299 T T F <--- 18 360-379 179 179 179 300 300 300 F F F 19 380-399 179 179 179 300 300 300 F F F 20 400-419 180 180 177 301 301 295 F F T <--- 21 420-439 180 180 179 301 301 300 F F T <--- 22 440-459 179 179 179 300 300 300 F F F 23 460-479 177 177 177 295 295 295 F F F 24 480-499 178 178 178 298 298 298 T T F <--- 25 500-519 178 178 178 296 296 296 F F F 26 520-539 173 173 173 290 290 290 F F F 27 540-559 178 178 178 298 298 298 T T F <--- 28 560-579 180 180 177 301 301 295 F F T <--- 29 580-599 178 178 178 296 296 296 F F F Max gen: 180, Max flush: 301 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 181 181 181 302 302 302 F F F 1 020-039 173 173 173 290 290 290 F F F 2 040-059 177 177 177 295 295 295 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 179 179 179 300 300 300 F F F 6 120-139 180 180 180 301 301 301 F F F 7 140-159 177 177 177 295 295 295 F F F 8 160-179 181 181 181 302 302 302 F F F 9 180-199 181 181 181 302 302 302 F F F 10 200-219 180 180 180 301 301 301 F F F 11 220-239 177 177 177 295 295 295 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 178 178 178 297 297 297 F F F 15 300-319 176 176 176 293 293 293 F F F 16 320-339 174 174 174 291 291 291 F F F 17 340-359 181 181 181 302 302 302 F F F 18 360-379 179 179 179 300 300 300 F F F 19 380-399 179 179 179 300 300 300 F F F 20 400-419 180 180 180 301 301 301 F F F 21 420-439 180 180 180 301 301 301 F F F 22 440-459 179 179 179 300 300 300 F F F 23 460-479 177 177 177 295 295 295 F F F 24 480-499 181 181 181 302 302 302 F F F 25 500-519 178 178 178 296 296 296 F F F 26 520-539 173 173 173 290 290 290 F F F 27 540-559 181 181 181 302 302 302 F F F 28 560-579 180 180 180 301 301 301 F F F 29 580-599 178 178 178 296 296 296 F F F Max gen: 181, Max flush: 302 Begin loop 91 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.0085299Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.05328178Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.054053653Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.054119477Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.054140575Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.054167452Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934} {"msg":"Crucible 1ae21a80-35b6-4705-899d-3e687df4b98c has session id: 9473bee2-d080-4d5a-8643-bd93b8b4613a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.054786169Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Upstairs opts: Upstairs UUID: 1ae21a80-35b6-4705-899d-3e687df4b98c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.054828875Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Crucible stats registered with UUID: 1ae21a80-35b6-4705-899d-3e687df4b98c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.054849333Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:59.054866132Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055077903Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"io task","client":"1","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.05511299Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"io task","client":"0","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055131739Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055367228Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"task":"crutest"} {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055401725Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934} {"msg":"1ae21a80-35b6-4705-899d-3e687df4b98c active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055420863Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055437032Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"client":"0","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.05545775Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"client":"1","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055475039Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"client":"2","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055491937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055509506Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"io task","client":"0","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055526864Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"io task","client":"1","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"ds_connection connected from Ok(127.0.0.1:54611)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055564871Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"ds_connection connected from Ok(127.0.0.1:35323)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055675541Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"io task","client":"1","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"ds_connection connected from Ok(127.0.0.1:45519)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.055718887Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"io task","client":"0","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.056859498Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"client":"1","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.056893895Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"client":"0","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.057505692Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.057538499Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.05855357Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"client":"2","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"[0]R flush_numbers[0..12]: [302, 290, 295, 285, 293, 300, 301, 295, 302, 302, 301, 295]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059049227Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"[0]R generation[0..12]: [181, 173, 177, 170, 176, 179, 180, 177, 181, 181, 180, 177]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059074435Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059099483Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"[1]R flush_numbers[0..12]: [302, 290, 295, 285, 293, 300, 301, 295, 302, 302, 301, 295]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059119641Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"[1]R generation[0..12]: [181, 173, 177, 170, 176, 179, 180, 177, 181, 181, 180, 177]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059138849Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059158488Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"[2]R flush_numbers[0..12]: [302, 290, 295, 285, 293, 300, 301, 295, 302, 302, 301, 295]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059177866Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"[2]R generation[0..12]: [181, 173, 177, 170, 176, 179, 180, 177, 181, 181, 180, 177]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059197144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059216783Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Max found gen is 182","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059233331Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:23:59.05925044Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Next flush: 303","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059272178Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059288486Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"":"downstairs","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059305615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059321314Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"client":"0","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059338732Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"client":"1","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059355691Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"client":"2","":"downstairs","session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059374719Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059390837Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"1ae21a80-35b6-4705-899d-3e687df4b98c is now active with session: 9473bee2-d080-4d5a-8643-bd93b8b4613a","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059408676Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059423794Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934,"session_id":"9473bee2-d080-4d5a-8643-bd93b8b4613a"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059442923Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934} {"msg":"Activated sub_volume 1ae21a80-35b6-4705-899d-3e687df4b98c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:23:59.059456962Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 210 len 40960 data: 13 14 14 13 14 14 13 15 17 18 02/30 Read block 186 len 32768 03/30 Read block 358 len 28672 04/30 Read block 164 len 16384 05/30 Write block 180 len 16384 data: 10 10 11 13 06/30 Write block 567 len 20480 data: 12 13 16 15 12 07/30 Write block 234 len 4096 data: 12 08/30 Read block 328 len 36864 09/30 Flush 10/30 Flush 11/30 Write block 575 len 40960 data: 13 11 10 11 11 11 9 9 10 11 12/30 Flush 13/30 Write block 282 len 8192 data: 17 16 14/30 Write block 13 len 20480 data: 18 15 15 13 13 15/30 Write block 556 len 40960 data: 13 8 5 5 8 5 7 9 11 10 16/30 Read block 394 len 36864 17/30 Read block 482 len 4096 18/30 Read block 77 len 20480 19/30 Read block 98 len 20480 20/30 Flush 21/30 Write block 164 len 16384 data: 16 15 17 13 22/30 Read block 429 len 28672 23/30 Read block 198 len 28672 24/30 Read block 179 len 20480 25/30 Write block 105 len 24576 data: 14 16 14 14 15 16 26/30 Read block 158 len 40960 27/30 Read block 368 len 8192 28/30 Write block 117 len 20480 data: 14 15 15 14 14 29/30 Write block 33 len 4096 data: 16 30/30 Write block 511 len 40960 data: 13 14 14 12 8 9 7 6 8 9 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:20 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1010 Acked Write 10 Done Done Done false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Write 5 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 4 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Read 10 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Write 5 Done Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 3 19 24 Done 18 17 1 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) Some(JobId(1009)) Downstairs last five completed: 1009 1008 1007 1006 1005 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:23:59.078959572Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1934} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 181 306 306 302 F F F <--- 1 020-039 173 173 173 290 290 290 T T F <--- 2 040-059 177 177 177 295 295 295 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 179 179 179 300 300 300 T T F <--- 6 120-139 180 180 180 301 301 301 T T F <--- 7 140-159 177 177 177 295 295 295 F F F 8 160-179 181 181 181 302 302 302 T T F <--- 9 180-199 182 182 182 303 303 303 F F F 10 200-219 182 182 182 303 303 303 F F F 11 220-239 182 182 182 303 303 303 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 182 182 178 306 306 297 F F F <--- 15 300-319 176 176 176 293 293 293 F F F 16 320-339 174 174 174 291 291 291 F F F 17 340-359 181 181 181 302 302 302 F F F 18 360-379 179 179 179 300 300 300 F F F 19 380-399 179 179 179 300 300 300 F F F 20 400-419 180 180 180 301 301 301 F F F 21 420-439 180 180 180 301 301 301 F F F 22 440-459 179 179 179 300 300 300 F F F 23 460-479 177 177 177 295 295 295 F F F 24 480-499 181 181 181 302 302 302 F F F 25 500-519 178 178 178 296 296 296 T T F <--- 26 520-539 173 173 173 290 290 290 T T F <--- 27 540-559 182 182 181 306 306 302 F F F <--- 28 560-579 182 182 182 306 306 303 F F T <--- 29 580-599 182 182 178 305 305 296 F F T <--- Max gen: 182, Max flush: 306 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 306 306 306 F F F 1 020-039 183 183 183 307 307 307 F F F 2 040-059 177 177 177 295 295 295 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 183 183 183 307 307 307 F F F 6 120-139 183 183 183 307 307 307 F F F 7 140-159 177 177 177 295 295 295 F F F 8 160-179 183 183 183 307 307 307 F F F 9 180-199 182 182 182 303 303 303 F F F 10 200-219 182 182 182 303 303 303 F F F 11 220-239 182 182 182 303 303 303 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 182 182 182 306 306 306 F F F 15 300-319 176 176 176 293 293 293 F F F 16 320-339 174 174 174 291 291 291 F F F 17 340-359 181 181 181 302 302 302 F F F 18 360-379 179 179 179 300 300 300 F F F 19 380-399 179 179 179 300 300 300 F F F 20 400-419 180 180 180 301 301 301 F F F 21 420-439 180 180 180 301 301 301 F F F 22 440-459 179 179 179 300 300 300 F F F 23 460-479 177 177 177 295 295 295 F F F 24 480-499 181 181 181 302 302 302 F F F 25 500-519 183 183 183 307 307 307 F F F 26 520-539 183 183 183 307 307 307 F F F 27 540-559 182 182 182 306 306 306 F F F 28 560-579 182 182 182 306 306 306 F F F 29 580-599 182 182 182 305 305 305 F F F Max gen: 183, Max flush: 307 Begin loop 92 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.485906277Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.532386377Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.533471212Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.533517458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.533533897Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.533556065Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942} {"msg":"Crucible d60abb10-8090-4519-9c8d-362b52b6a611 has session id: 69180bdc-2cbe-4356-9c77-0cbb1880c68b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534178721Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Upstairs opts: Upstairs UUID: d60abb10-8090-4519-9c8d-362b52b6a611, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534249155Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Crucible stats registered with UUID: d60abb10-8090-4519-9c8d-362b52b6a611","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534287501Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:03.534320918Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.53452666Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"0","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534577366Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534610403Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"2","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534769949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"task":"crutest"} {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534807236Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942} {"msg":"d60abb10-8090-4519-9c8d-362b52b6a611 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534836863Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534857671Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"client":"0","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534884699Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534922286Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"client":"2","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.534963382Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"2","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.535002399Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"0","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.535038996Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"ds_connection connected from Ok(127.0.0.1:50955)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.535076282Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"2","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"ds_connection connected from Ok(127.0.0.1:56734)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.535212091Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"0","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:03.535262776Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:03.535287944Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:03.535312512Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.535420562Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.53545149Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.536464111Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"client":"0","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.536499618Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"client":"2","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.537196048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:03.537235484Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.536870023Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"ds_connection connected from Ok(127.0.0.1:32876)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.537061616Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.538561986Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"[0]R flush_numbers[0..12]: [306, 307, 295, 285, 293, 307, 307, 295, 307, 303, 303, 303]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539180042Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"[0]R generation[0..12]: [182, 183, 177, 170, 176, 183, 183, 177, 183, 182, 182, 182]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539211559Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539238187Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"[1]R flush_numbers[0..12]: [306, 307, 295, 285, 293, 307, 307, 295, 307, 303, 303, 303]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539262474Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"[1]R generation[0..12]: [182, 183, 177, 170, 176, 183, 183, 177, 183, 182, 182, 182]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539292582Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.5393181Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"[2]R flush_numbers[0..12]: [306, 307, 295, 285, 293, 307, 307, 295, 307, 303, 303, 303]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539342258Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"[2]R generation[0..12]: [182, 183, 177, 170, 176, 183, 183, 177, 183, 182, 182, 182]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539365556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539389933Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539410452Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.53943186Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Next flush: 308","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539452408Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539472776Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"":"downstairs","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"No downstairs reconciliation required","v":Wait for a query_work_queue command to finish before sending IO 0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539504843Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539524371Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"client":"0","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.53954552Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"client":"1","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539566578Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"client":"2","":"downstairs","session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539588596Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539608774Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"d60abb10-8090-4519-9c8d-362b52b6a611 is now active with session: 69180bdc-2cbe-4356-9c77-0cbb1880c68b","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539633622Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539649261Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942,"session_id":"69180bdc-2cbe-4356-9c77-0cbb1880c68b"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539665559Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942} {"msg":"Activated sub_volume d60abb10-8090-4519-9c8d-362b52b6a611","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:13.539679668Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 531 len 12288 data: 17 15 15 02/30 Read block 138 len 12288 03/30 Write block 538 len 40960 data: 13 11 14 14 16 14 13 10 10 9 04/30 Write block 217 len 4096 data: 16 05/30 Flush 06/30 Write block 198 len 16384 data: 21 20 19 18 07/30 Read block 548 len 8192 08/30 Write block 31 len 4096 data: 14 09/30 Write block 323 len 12288 data: 14 14 13 10/30 Write block 342 len 40960 data: 13 12 14 14 12 16 17 17 16 12 11/30 Read block 62 len 32768 12/30 Read block 496 len 32768 13/30 Read block 42 len 40960 14/30 Write block 398 len 24576 data: 12 12 9 13 13 15 15/30 Read block 82 len 40960 16/30 Read block 261 len 28672 17/30 Read block 353 len 4096 18/30 Write block 179 len 24576 data: 11 11 11 12 14 10 19/30 Write block 108 len 28672 data: 15 16 17 16 14 17 15 20/30 Write block 346 len 36864 data: 13 17 18 18 17 13 15 16 16 21/30 Read block 108 len 20480 22/30 Write block 427 len 32768 data: 17 15 15 16 18 18 17 15 23/30 Write block 541 len 12288 data: 15 17 15 24/30 Flush 25/30 Flush 26/30 Write block 178 len 4096 data: 12 27/30 Write block 122 len 16384 data: 14 13 12 12 28/30 Write block 458 len 24576 data: 11 12 11 11 12 14 29/30 Write block 394 len 12288 data: 8 11 13 30/30 Write block 563 len 40960 data: 10 12 11 13 13 14 17 16 13 13 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Read 3 Done Sent Done false 1002 Acked Write 10 Done Sent Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 4 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Write 1 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Write 10 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 8 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Write 6 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 6 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 30 3 36 Done 27 0 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:13.558093723Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1942} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 306 306 306 F F F 1 020-039 184 183 184 309 307 309 F F F <--- 2 040-059 177 177 177 295 295 295 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 184 183 184 309 307 309 F F F <--- 6 120-139 183 183 183 307 307 307 T F T <--- 7 140-159 177 177 177 295 295 295 F F F 8 160-179 184 183 184 309 307 309 T F T <--- 9 180-199 184 182 184 309 303 309 F F F <--- 10 200-219 184 182 184 309 303 309 F F F <--- 11 220-239 182 182 182 303 303 303 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 182 182 182 306 306 306 F F F 15 300-319 176 176 176 293 293 293 F F F 16 320-339 184 174 184 309 291 309 F F F <--- 17 340-359 184 181 184 309 302 309 F F F <--- 18 360-379 179 179 179 300 300 300 F F F 19 380-399 184 179 184 309 300 309 T F T <--- 20 400-419 184 180 184 309 301 309 F F F <--- 21 420-439 184 180 184 309 301 309 F F F <--- 22 440-459 179 179 179 300 300 300 T F T <--- 23 460-479 177 177 177 295 295 295 T F T <--- 24 480-499 181 181 181 302 302 302 F F F 25 500-519 183 183 183 307 307 307 F F F 26 520-539 184 183 184 308 307 308 F F F <--- 27 540-559 184 182 184 309 306 309 F F F <--- 28 560-579 182 182 182 306 306 306 T F T <--- 29 580-599 182 182 182 305 305 305 F F F Max gen: 184, Max flush: 309 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 306 306 306 F F F 1 020-039 184 184 184 309 309 309 F F F 2 040-059 177 177 177 295 295 295 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 184 184 184 309 309 309 F F F 6 120-139 185 185 185 310 310 310 F F F 7 140-159 177 177 177 295 295 295 F F F 8 160-179 185 185 185 310 310 310 F F F 9 180-199 184 184 184 309 309 309 F F F 10 200-219 184 184 184 309 309 309 F F F 11 220-239 182 182 182 303 303 303 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 182 182 182 306 306 306 F F F 15 300-319 176 176 176 293 293 293 F F F 16 320-339 184 184 184 309 309 309 F F F 17 340-359 184 184 184 309 309 309 F F F 18 360-379 179 179 179 300 300 300 F F F 19 380-399 185 185 185 310 310 310 F F F 20 400-419 184 184 184 309 309 309 F F F 21 420-439 184 184 184 309 309 309 F F F 22 440-459 185 185 185 310 310 310 F F F 23 460-479 185 185 185 310 310 310 F F F 24 480-499 181 181 181 302 302 302 F F F 25 500-519 183 183 183 307 307 307 F F F 26 520-539 184 184 184 308 308 308 F F F 27 540-559 184 184 184 309 309 309 F F F 28 560-579 185 185 185 310 310 310 F F F 29 580-599 182 182 182 305 305 305 F F F Max gen: 185, Max flush: 310 Begin loop 93 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.014302314Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.058898758Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.059649542Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.059707817Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.059727796Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.059754943Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950} {"msg":"Crucible c11b3449-5b4a-4a0f-b158-3311b81b9626 has session id: f0bb8406-d238-4725-82e4-235d71a47bb0","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.06024376Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Upstairs opts: Upstairs UUID: c11b3449-5b4a-4a0f-b158-3311b81b9626, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060297436Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Crucible stats registered with UUID: c11b3449-5b4a-4a0f-b158-3311b81b9626","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060331693Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:18.06036136Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060572372Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"io task","client":"1","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060620718Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060652045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"io task","client":"2","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060803112Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"task":"crutest"} {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060832029Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950} {"msg":"c11b3449-5b4a-4a0f-b158-3311b81b9626 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060888044Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060920081Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"client":"0","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.06093957Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"client":"1","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060957048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"client":"2","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.060984776Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"io task","client":"2","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.061018483Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.06105547Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"io task","client":"1","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"ds_connection connected from Ok(127.0.0.1:56133)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.061087977Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"ds_connection connected from Ok(127.0.0.1:61961)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.061119394Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"io task","client":"2","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"ds_connection connected from Ok(127.0.0.1:56429)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.061152951Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"io task","client":"1","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.062247236Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"client":"0","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.062370255Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"client":"1","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.06289382Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.06300535Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.063538773Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"client":"2","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"[0]R flush_numbers[0..12]: [306, 309, 295, 285, 293, 309, 310, 295, 310, 309, 309, 303]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064059058Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"[0]R generation[0..12]: [182, 184, 177, 170, 176, 184, 185, 177, 185, 184, 184, 182]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064084436Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064105214Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"[1]R flush_numbers[0..12]: [306, 309, 295, 285, 293, 309, 310, 295, 310, 309, 309, 303]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064125612Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"[1]R generation[0..12]: [182, 184, 177, 170, 176, 184, 185, 177, 185, 184, 184, 182]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.06414492Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064164879Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"[2]R flush_numbers[0..12]: [306, 309, 295, 285, 293, 309, 310, 295, 310, 309, 309, 303]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064184027Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"[2]R generation[0..12]: [182, 184, 177, 170, 176, 184, 185, 177, 185, 184, 184, 182]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064202815Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064222254Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Max found gen is 186","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064238782Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Generation requested: 186 >= found:186","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-12-24T00:24:18.064259421Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Next flush: 311","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064284159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064300497Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"":"downstairs","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064317645Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064333254Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"client":"0","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064350243Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"client":"1","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064367161Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"client":"2","":"downstairs","session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.06438417Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064400638Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"c11b3449-5b4a-4a0f-b158-3311b81b9626 is now active with session: f0bb8406-d238-4725-82e4-235d71a47bb0","v":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064423346Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064438915Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950,"session_id":"f0bb8406-d238-4725-82e4-235d71a47bb0"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064455084Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950} {"msg":"Activated sub_volume c11b3449-5b4a-4a0f-b158-3311b81b9626","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:18.064469322Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 454 len 32768 data: 13 13 12 11 12 13 12 12 02/30 Read block 452 len 32768 03/30 Read block 431 len 36864 04/30 Read block 590 len 8192 05/30 Read block 192 len 16384 06/30 Read block 171 len 24576 07/30 Flush 08/30 Write block 429 len 40960 data: 16 17 19 19 18 16 16 16 16 16 09/30 Write block 517 len 16384 data: 8 7 9 10 10/30 Write block 179 len 16384 data: 12 12 12 13 11/30 Read block 244 len 8192 12/30 Write block 469 len 40960 data: 11 11 13 11 9 10 10 7 6 6 13/30 Read block 587 len 4096 14/30 Write block 312 len 16384 data: 12 11 11 12 15/30 Read block 512 len 20480 16/30 Flush 17/30 Write block 181 len 12288 data: 13 14 15 18/30 Write block 591 len 12288 data: 16 18 18 19/30 Write block 380 len 28672 data: 15 16 14 14 19 15 14 20/30 Write block 466 len 12288 data: 12 12 10 21/30 Flush 22/30 Write block 446 len 32768 data: 17 19 17 18 18 16 14 12 23/30 Read block 188 len 24576 24/30 Read block 385 len 40960 25/30 Write block 475 len 4096 data: 11 26/30 Write block 443 len 12288 data: 18 19 18 27/30 Write block 41 len 16384 data: 10 9 12 12 28/30 Read block 461 len 28672 29/30 Read block 400 len 28672 30/30 Read block 402 len 24576 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 8 Done Done Sent false 1002 Acked Read 9 Done Done Sent false 1003 Acked Read 2 Done Done Sent false 1004 Acked Read 4 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Write 4 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 4 Done Done Sent false 1014 Acked Read 5 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Write 3 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Read 6 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 29 29 Done 30 30 1 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:18.083703206Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1950} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 306 306 306 F F F 1 020-039 184 184 184 309 309 309 F F F 2 040-059 177 177 177 295 295 295 T T F <--- 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 184 184 184 309 309 309 F F F 6 120-139 185 185 185 310 310 310 F F F 7 140-159 177 177 177 295 295 295 F F F 8 160-179 186 186 185 312 312 310 F F F <--- 9 180-199 186 186 184 313 313 309 F F F <--- 10 200-219 184 184 184 309 309 309 F F F 11 220-239 182 182 182 303 303 303 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 182 182 182 306 306 306 F F F 15 300-319 186 186 176 312 312 293 F F F <--- 16 320-339 184 184 184 309 309 309 F F F 17 340-359 184 184 184 309 309 309 F F F 18 360-379 179 179 179 300 300 300 F F F 19 380-399 186 186 185 313 313 310 F F F <--- 20 400-419 184 184 184 309 309 309 F F F 21 420-439 186 186 184 312 312 309 F F F <--- 22 440-459 186 186 185 311 311 310 T T T <--- 23 460-479 186 186 185 313 313 310 T T T <--- 24 480-499 181 181 181 302 302 302 F F F 25 500-519 186 186 183 312 312 307 F F F <--- 26 520-539 186 186 184 312 312 308 F F F <--- 27 540-559 184 184 184 309 309 309 F F F 28 560-579 185 185 185 310 310 310 F F F 29 580-599 186 186 182 313 313 305 F F F <--- Max gen: 186, Max flush: 313 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 306 306 306 F F F 1 020-039 184 184 184 309 309 309 F F F 2 040-059 187 187 187 314 314 314 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 184 184 184 309 309 309 F F F 6 120-139 185 185 185 310 310 310 F F F 7 140-159 177 177 177 295 295 295 F F F 8 160-179 186 186 186 312 312 312 F F F 9 180-199 186 186 186 313 313 313 F F F 10 200-219 184 184 184 309 309 309 F F F 11 220-239 182 182 182 303 303 303 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 182 182 182 306 306 306 F F F 15 300-319 186 186 186 312 312 312 F F F 16 320-339 184 184 184 309 309 309 F F F 17 340-359 184 184 184 309 309 309 F F F 18 360-379 179 179 179 300 300 300 F F F 19 380-399 186 186 186 313 313 313 F F F 20 400-419 184 184 184 309 309 309 F F F 21 420-439 186 186 186 312 312 312 F F F 22 440-459 187 187 187 314 314 314 F F F 23 460-479 187 187 187 314 314 314 F F F 24 480-499 181 181 181 302 302 302 F F F 25 500-519 186 186 186 312 312 312 F F F 26 520-539 186 186 186 312 312 312 F F F 27 540-559 184 184 184 309 309 309 F F F 28 560-579 185 185 185 310 310 310 F F F 29 580-599 186 186 186 313 313 313 F F F Max gen: 187, Max flush: 314 Begin loop 94 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.483627242Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:22.528471444Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.528881088Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.559766777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.560444138Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.560500333Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.560520741Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.560548509Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958} {"msg":"Crucible d71b670e-6155-498d-9011-334da5ffffe8 has session id: ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561028047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Upstairs opts: Upstairs UUID: d71b670e-6155-498d-9011-334da5ffffe8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561090101Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Crucible stats registered with UUID: d71b670e-6155-498d-9011-334da5ffffe8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561123438Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:22.561141937Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561311452Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"io task","client":"1","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561348889Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"io task","client":"0","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561367777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561528713Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"task":"crutest"} {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561552941Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958} {"msg":"d71b670e-6155-498d-9011-334da5ffffe8 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561623265Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561647423Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"client":"0","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561665621Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"client":"1","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561689519Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"client":"2","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561721006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561750324Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"io task","client":"0","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561768652Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"io task","client":"1","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:33447)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.56179793Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:33402)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561833557Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"io task","client":"1","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"ds_connection connected from Ok(127.0.0.1:60399)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.561870573Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"io task","client":"0","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.563207667Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"client":"1","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.563243874Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"client":"2","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.563265172Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"client":"0","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.563834442Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.563930714Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"[0]R flush_numbers[0..12]: [306, 309, 314, 285, 293, 309, 310, 295, 312, 313, 309, 303]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564042234Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"[0]R generation[0..12]: [182, 184, 187, 170, 176, 184, 185, 177, 186, 186, 184, 182]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564069842Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.56409071Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"[1]R flush_numbers[0..12]: [306, 309, 314, 285, 293, 309, 310, 295, 312, 313, 309, 303]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564110018Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"[1]R generation[0..12]: [182, 184, 187, 170, 176, 184, 185, 177, 186, 186, 184, 182]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564129147Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564148775Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"[2]R flush_numbers[0..12]: [306, 309, 314, 285, 293, 309, 310, 295, 312, 313, 309, 303]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564167913Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"[2]R generation[0..12]: [182, 184, 187, 170, 176, 184, 185, 177, 186, 186, 184, 182]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564186742Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.56420651Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Max found gen is 188","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564223048Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564240357Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Next flush: 315","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564256916Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564273154Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"":"downstairs","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564290313Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-12-24T00:24:22.564305851Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"client":"0","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564328389Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"client":"1","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564345518Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"client":"2","":"downstairs","session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564362786Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564379205Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"d71b670e-6155-498d-9011-334da5ffffe8 is now active with session: ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564397003Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564412462Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958,"session_id":"ad4cf7a2-9d59-44b9-a0f6-9bcf2847f8ee"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.56442859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958} {"msg":"Activated sub_volume d71b670e-6155-498d-9011-334da5ffffe8","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:22.564443139Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 146 len 12288 data: 11 10 12 02/30 Write block 283 len 8192 data: 17 14 03/30 Read block 25 len 24576 04/30 Read block 496 len 12288 05/30 Read block 4 len 4096 06/30 Read block 27 len 28672 07/30 Read block 283 len 8192 08/30 Read block 281 len 40960 09/30 Write block 532 len 16384 data: 16 16 14 12 10/30 Write block 371 len 8192 data: 9 10 11/30 Write block 483 len 4096 data: 17 12/30 Write block 467 len 16384 data: 13 11 12 12 13/30 Read block 394 len 8192 14/30 Read block 40 len 32768 15/30 Write block 381 len 36864 data: 17 15 15 20 16 15 14 17 15 16/30 Write block 491 len 36864 data: 13 16 14 16 17 16 12 12 13 17/30 Flush 18/30 Write block 136 len 12288 data: 12 12 15 19/30 Read block 144 len 20480 20/30 Read block 557 len 32768 21/30 Read block 307 len 36864 22/30 Read block 543 len 12288 23/30 Write block 460 len 40960 data: 13 13 13 15 17 14 13 14 12 13 24/30 Write block 346 len 36864 data: 14 18 19 19 18 14 16 17 17 25/30 Read block 203 len 8192 26/30 Read block 562 len 40960 27/30 Write block 217 len 20480 data: 17 18 19 18 18 28/30 Write block 463 len 4096 data: 16 29/30 Write block 462 len 8192 data: 14 17 30/30 Read block 297 len 12288 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 6 Sent Done Done false 1003 Acked Read 3 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Read 7 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Read 8 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Write 2 Sent Done Sent false 1029 Acked Read 3 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 2 30 Done 2 30 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:22.582697989Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1958} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 306 306 306 F F F 1 020-039 184 184 184 309 309 309 F F F 2 040-059 187 187 187 314 314 314 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 184 184 184 309 309 309 F F F 6 120-139 185 185 185 310 310 310 F T T <--- 7 140-159 177 188 188 295 315 315 T F F <--- 8 160-179 186 186 186 312 312 312 F F F 9 180-199 186 186 186 313 313 313 F F F 10 200-219 184 184 184 309 309 309 F T T <--- 11 220-239 182 182 182 303 303 303 F T T <--- 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 182 188 188 306 315 315 T F F <--- 15 300-319 186 186 186 312 312 312 F F F 16 320-339 184 184 184 309 309 309 F F F 17 340-359 184 184 184 309 309 309 F T T <--- 18 360-379 179 188 188 300 315 315 F F F <--- 19 380-399 186 188 188 313 315 315 F F F <--- 20 400-419 184 184 184 309 309 309 F F F 21 420-439 186 186 186 312 312 312 F F F 22 440-459 187 187 187 314 314 314 F F F 23 460-479 187 188 188 314 315 315 F T T <--- 24 480-499 181 188 188 302 315 315 F F F <--- 25 500-519 186 186 186 312 312 312 F F F 26 520-539 186 188 188 312 315 315 F F F <--- 27 540-559 184 184 184 309 309 309 F F F 28 560-579 185 185 185 310 310 310 F F F 29 580-599 186 186 186 313 313 313 F F F Max gen: 188, Max flush: 315 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 306 306 306 F F F 1 020-039 184 184 184 309 309 309 F F F 2 040-059 187 187 187 314 314 314 F F F 3 060-079 170 170 170 285 285 285 F F F 4 080-099 176 176 176 293 293 293 F F F 5 100-119 184 184 184 309 309 309 F F F 6 120-139 189 189 189 316 316 316 F F F 7 140-159 188 188 188 315 315 315 F F F 8 160-179 186 186 186 312 312 312 F F F 9 180-199 186 186 186 313 313 313 F F F 10 200-219 189 189 189 316 316 316 F F F 11 220-239 189 189 189 316 316 316 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 188 188 188 315 315 315 F F F 15 300-319 186 186 186 312 312 312 F F F 16 320-339 184 184 184 309 309 309 F F F 17 340-359 189 189 189 316 316 316 F F F 18 360-379 188 188 188 315 315 315 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 184 184 184 309 309 309 F F F 21 420-439 186 186 186 312 312 312 F F F 22 440-459 187 187 187 314 314 314 F F F 23 460-479 189 189 189 316 316 316 F F F 24 480-499 188 188 188 315 315 315 F F F 25 500-519 186 186 186 312 312 312 F F F 26 520-539 188 188 188 315 315 315 F F F 27 540-559 184 184 184 309 309 309 F F F 28 560-579 185 185 185 310 310 310 F F F 29 580-599 186 186 186 313 313 313 F F F Max gen: 189, Max flush: 316 Begin loop 95 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.017639642Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.064190426Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.06494233Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.064990666Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.065015284Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.065045281Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966} {"msg":"Crucible f39ba47a-d332-4eb3-b704-1093c8a9e127 has session id: c9edd9d3-1023-4009-8342-bb1fa647734e","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.065592803Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Upstairs opts: Upstairs UUID: f39ba47a-d332-4eb3-b704-1093c8a9e127, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.065652068Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Crucible stats registered with UUID: f39ba47a-d332-4eb3-b704-1093c8a9e127","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.065691065Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:27.065721082Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.065883298Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.065923785Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"io task","client":"0","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.065943363Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"io task","client":"2","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066123197Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"task":"crutest"} {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066155594Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966} {"msg":"f39ba47a-d332-4eb3-b704-1093c8a9e127 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066231748Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066253556Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"client":"0","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066271994Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"client":"1","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066289183Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"client":"2","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066306201Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"io task","client":"2","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.06632384Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"io task","client":"0","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066418982Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"ds_connection connected from Ok(127.0.0.1:54640)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066479466Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"io task","client":"2","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"ds_connection connected from Ok(127.0.0.1:37370)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066520083Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"io task","client":"0","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"ds_connection connected from Ok(127.0.0.1:52549)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.066625004Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.067638145Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"client":"0","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.067752485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"client":"1","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.068314476Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.068397739Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.068914244Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"client":"2","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"[0]R flush_numbers[0..12]: [306, 309, 314, 285, 293, 309, 316, 315, 312, 313, 316, 316]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069434429Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"[0]R generation[0..12]: [182, 184, 187, 170, 176, 184, 189, 188, 186, 186, 189, 189]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069458797Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069478925Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"[1]R flush_numbers[0..12]: [306, 309, 314, 285, 293, 309, 316, 315, 312, 313, 316, 316]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069498123Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"[1]R generation[0..12]: [182, 184, 187, 170, 176, 184, 189, 188, 186, 186, 189, 189]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069517052Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.06953668Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"[2]R flush_numbers[0..12]: [306, 309, 314, 285, 293, 309, 316, 315, 312, 313, 316, 316]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069555718Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"[2]R generation[0..12]: [182, 184, 187, 170, 176, 184, 189, 188, 186, 186, 189, 189]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069577946Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069597845Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069614373Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069631382Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Next flush: 317","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.06964752Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069663849Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"":"downstairs","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069680847Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069696046Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"client":"0","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069713255Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"client":"1","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} Wait for a query_work_queue command to finish before sending IO {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069735833Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"client":"2","":"downstairs","session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069753031Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.06976949Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"f39ba47a-d332-4eb3-b704-1093c8a9e127 is now active with session: c9edd9d3-1023-4009-8342-bb1fa647734e","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069787368Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069802827Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966,"session_id":"c9edd9d3-1023-4009-8342-bb1fa647734e"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069819045Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966} {"msg":"Activated sub_volume f39ba47a-d332-4eb3-b704-1093c8a9e127","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:27.069833024Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 312 len 32768 data: 13 12 12 13 12 12 10 10 02/30 Read block 67 len 12288 03/30 Write block 94 len 16384 data: 15 17 15 17 04/30 Read block 543 len 28672 05/30 Read block 437 len 32768 06/30 Write block 299 len 8192 data: 16 16 07/30 Read block 220 len 12288 08/30 Read block 536 len 32768 09/30 Flush 10/30 Write block 411 len 36864 data: 13 9 10 12 13 12 10 12 14 11/30 Write block 12 len 40960 data: 17 19 16 16 14 14 14 15 11 12 12/30 Read block 538 len 4096 13/30 Read block 213 len 24576 14/30 Write block 331 len 40960 data: 15 12 12 11 10 10 12 9 11 17 15/30 Read block 380 len 28672 16/30 Write block 453 len 8192 data: 13 14 17/30 Write block 223 len 36864 data: 12 14 10 9 10 11 11 14 16 18/30 Read block 312 len 20480 19/30 Read block 411 len 12288 20/30 Write block 558 len 28672 data: 6 6 9 6 8 11 13 21/30 Write block 154 len 32768 data: 13 12 14 13 11 14 18 22 22/30 Flush 23/30 Write block 118 len 8192 data: 16 16 24/30 Write block 292 len 40960 data: 12 11 15 14 12 11 11 17 17 16 25/30 Flush 26/30 Write block 571 len 28672 data: 14 14 14 13 14 12 11 27/30 Write block 327 len 36864 data: 13 14 15 13 16 13 13 12 11 28/30 Write block 66 len 12288 data: 15 17 18 29/30 Read block 195 len 32768 30/30 Write block 550 len 8192 data: 14 12 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Sent false 1001 Acked Read 3 Done Done Sent false 1002 Acked Write 4 Done Done Sent false 1003 Acked Read 7 Done Done Sent false 1004 Acked Read 8 Done Done Sent false 1005 Acked Write 2 Done Done Sent false 1006 Acked Read 3 Done Done Sent false 1007 Acked Read 8 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 8 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Write 3 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:27.09041536Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1966} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 190 190 182 318 318 306 F F F <--- 1 020-039 190 190 184 318 318 309 F F F <--- 2 040-059 187 187 187 314 314 314 F F F 3 060-079 170 170 170 285 285 285 T T F <--- 4 080-099 190 190 176 317 317 293 F F F <--- 5 100-119 190 190 184 319 319 309 F F F <--- 6 120-139 189 189 189 316 316 316 F F F 7 140-159 190 190 188 318 318 315 F F F <--- 8 160-179 190 190 186 318 318 312 F F F <--- 9 180-199 186 186 186 313 313 313 F F F 10 200-219 189 189 189 316 316 316 F F F 11 220-239 190 190 189 318 318 316 F F F <--- 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 190 190 188 319 319 315 F F F <--- 15 300-319 190 190 186 319 319 312 F F F <--- 16 320-339 190 190 184 318 318 309 T T F <--- 17 340-359 190 190 189 318 318 316 F F F <--- 18 360-379 188 188 188 315 315 315 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 190 190 184 318 318 309 F F F <--- 21 420-439 186 186 186 312 312 312 F F F 22 440-459 190 190 187 318 318 314 F F F <--- 23 460-479 189 189 189 316 316 316 F F F 24 480-499 188 188 188 315 315 315 F F F 25 500-519 186 186 186 312 312 312 F F F 26 520-539 188 188 188 315 315 315 F F F 27 540-559 190 190 184 318 318 309 T T F <--- 28 560-579 190 190 185 318 318 310 T T F <--- 29 580-599 186 186 186 313 313 313 F F F Max gen: 190, Max flush: 319 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 95 Verifying data now No change after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 190 190 190 318 318 318 F F F 1 020-039 190 190 190 318 318 318 F F F 2 040-059 187 187 187 314 314 314 F F F 3 060-079 191 191 191 320 320 320 F F F 4 080-099 190 190 190 317 317 317 F F F 5 100-119 190 190 190 319 319 319 F F F 6 120-139 189 189 189 316 316 316 F F F 7 140-159 190 190 190 318 318 318 F F F 8 160-179 190 190 190 318 318 318 F F F 9 180-199 186 186 186 313 313 313 F F F 10 200-219 189 189 189 316 316 316 F F F 11 220-239 190 190 190 318 318 318 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 181 181 181 302 302 302 F F F 14 280-299 190 190 190 319 319 319 F F F 15 300-319 190 190 190 319 319 319 F F F 16 320-339 191 191 191 320 320 320 F F F 17 340-359 190 190 190 318 318 318 F F F 18 360-379 188 188 188 315 315 315 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 190 190 190 318 318 318 F F F 21 420-439 186 186 186 312 312 312 F F F 22 440-459 190 190 190 318 318 318 F F F 23 460-479 189 189 189 316 316 316 F F F 24 480-499 188 188 188 315 315 315 F F F 25 500-519 186 186 186 312 312 312 F F F 26 520-539 188 188 188 315 315 315 F F F 27 540-559 191 191 191 320 320 320 F F F 28 560-579 191 191 191 320 320 320 F F F 29 580-599 186 186 186 313 313 313 F F F Max gen: 191, Max flush: 320 Begin loop 96 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.513810882Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.558447442Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.559327375Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.559375691Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.559392489Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.559414657Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974} {"msg":"Crucible 982357c7-4b76-41fc-9d20-e6083ea4854c has session id: af8ad9aa-bff3-447b-9295-74d29fcd9954","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560045512Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Upstairs opts: Upstairs UUID: 982357c7-4b76-41fc-9d20-e6083ea4854c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560116636Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Crucible stats registered with UUID: 982357c7-4b76-41fc-9d20-e6083ea4854c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560154963Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:31.56018802Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560441028Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560477765Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"io task","client":"1","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560497693Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"io task","client":"0","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560678207Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"task":"crutest"} {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560705485Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974} {"msg":"982357c7-4b76-41fc-9d20-e6083ea4854c active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560735272Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560751341Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"client":"0","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560768899Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"client":"1","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560786218Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"client":"2","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560803136Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560820515Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"io task","client":"0","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.560849452Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"io task","client":"1","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"ds_connection connected from Ok(127.0.0.1:65416)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.56110693Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"io task","client":"1","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"ds_connection connected from Ok(127.0.0.1:34734)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.561151236Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"ds_connection connected from Ok(127.0.0.1:38524)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.561185843Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"io task","client":"0","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.562337103Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"client":"2","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.56237257Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"client":"0","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.562774415Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"client":"1","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.56294376Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563020863Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"[0]R flush_numbers[0..12]: [318, 318, 314, 320, 317, 319, 316, 318, 318, 313, 316, 318]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563350854Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"[0]R generation[0..12]: [190, 190, 187, 191, 190, 190, 189, 190, 190, 186, 189, 190]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563380322Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.56340251Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"[1]R flush_numbers[0..12]: [318, 318, 314, 320, 317, 319, 316, 318, 318, 313, 316, 318]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563421978Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"[1]R generation[0..12]: [190, 190, 187, 191, 190, 190, 189, 190, 190, 186, 189, 190]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563441317Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563461095Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"[2]R flush_numbers[0..12]: [318, 318, 314, 320, 317, 319, 316, 318, 318, 313, 316, 318]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563480153Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"[2]R generation[0..12]: [190, 190, 187, 191, 190, 190, 189, 190, 190, 186, 189, 190]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563499121Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.56351892Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563535398Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563604502Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Next flush: 321","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563623191Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563640919Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"":"downstairs","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563661187Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563677006Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"client":"0","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563694044Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"client":"1","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563711133Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"client":"2","":"downstairs","session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563728361Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563749779Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"982357c7-4b76-41fc-9d20-e6083ea4854c is now active with session: af8ad9aa-bff3-447b-9295-74d29fcd9954","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563768068Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563783577Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974,"session_id":"af8ad9aa-bff3-447b-9295-74d29fcd9954"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563799605Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974} {"msg":"Activated sub_volume 982357c7-4b76-41fc-9d20-e6083ea4854c","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:31.563813604Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 533 len 8192 data: 17 15 02/30 Write block 576 len 8192 data: 13 12 03/30 Write block 261 len 32768 data: 13 15 12 12 14 14 15 14 04/30 Write block 80 len 28672 data: 13 13 15 18 13 11 11 05/30 Read block 237 len 12288 06/30 Read block 202 len 12288 07/30 Read block 249 len 16384 08/30 Write block 266 len 4096 data: 15 09/30 Read block 88 len 16384 10/30 Read block 357 len 40960 11/30 Write block 518 len 24576 data: 8 10 11 11 13 11 12/30 Read block 572 len 24576 13/30 Read block 268 len 12288 14/30 Write block 489 len 24576 data: 9 8 14 17 15 17 15/30 Write block 204 len 28672 data: 16 18 16 14 13 16 14 16/30 Read block 83 len 40960 17/30 Write block 17 len 32768 data: 15 15 16 12 13 11 12 13 18/30 Write block 273 len 36864 data: 16 16 15 18 17 15 13 15 17 19/30 Write block 26 len 24576 data: 13 12 10 12 12 15 20/30 Read block 420 len 20480 21/30 Read block 565 len 28672 22/30 Flush 23/30 Flush 24/30 Write block 213 len 8192 data: 14 15 25/30 Read block 489 len 28672 26/30 Read block 214 len 24576 27/30 Read block 398 len 28672 28/30 Write block 271 len 32768 data: 14 14 17 17 16 19 18 16 29/30 Write block 260 len 36864 data: 13 14 16 13 13 15 16 16 15 30/30 Write block 575 len 8192 data: 15 14 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Write 8 Done Sent Done false 1003 Acked Write 7 Done Sent Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Read 3 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Write 1 Done Sent Done false 1008 Acked Read 4 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Write 8 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 7 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 2 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Read 7 Done Sent Done false 1027 Acked Write 8 Done Sent Done false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 28 2 32 Done 28 2 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:31.581558458Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1974} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 190 192 321 318 321 F F F <--- 1 020-039 192 190 192 321 318 321 F F F <--- 2 040-059 187 187 187 314 314 314 F F F 3 060-079 191 191 191 320 320 320 F F F 4 080-099 192 190 192 321 317 321 F F F <--- 5 100-119 190 190 190 319 319 319 F F F 6 120-139 189 189 189 316 316 316 F F F 7 140-159 190 190 190 318 318 318 F F F 8 160-179 190 190 190 318 318 318 F F F 9 180-199 186 186 186 313 313 313 F F F 10 200-219 192 189 192 321 316 321 T F T <--- 11 220-239 190 190 190 318 318 318 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 192 181 192 321 302 321 T F T <--- 14 280-299 192 190 192 321 319 321 F F F <--- 15 300-319 190 190 190 319 319 319 F F F 16 320-339 191 191 191 320 320 320 F F F 17 340-359 190 190 190 318 318 318 F F F 18 360-379 188 188 188 315 315 315 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 190 190 190 318 318 318 F F F 21 420-439 186 186 186 312 312 312 F F F 22 440-459 190 190 190 318 318 318 F F F 23 460-479 189 189 189 316 316 316 F F F 24 480-499 192 188 192 321 315 321 F F F <--- 25 500-519 192 186 192 321 312 321 F F F <--- 26 520-539 192 188 192 321 315 321 F T F <--- 27 540-559 191 191 191 320 320 320 F F F 28 560-579 192 191 192 321 320 321 T T T <--- 29 580-599 186 186 186 313 313 313 F F F Max gen: 192, Max flush: 321 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 321 321 321 F F F 1 020-039 192 192 192 321 321 321 F F F 2 040-059 187 187 187 314 314 314 F F F 3 060-079 191 191 191 320 320 320 F F F 4 080-099 192 192 192 321 321 321 F F F 5 100-119 190 190 190 319 319 319 F F F 6 120-139 189 189 189 316 316 316 F F F 7 140-159 190 190 190 318 318 318 F F F 8 160-179 190 190 190 318 318 318 F F F 9 180-199 186 186 186 313 313 313 F F F 10 200-219 193 193 193 322 322 322 F F F 11 220-239 190 190 190 318 318 318 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 193 193 193 322 322 322 F F F 14 280-299 192 192 192 321 321 321 F F F 15 300-319 190 190 190 319 319 319 F F F 16 320-339 191 191 191 320 320 320 F F F 17 340-359 190 190 190 318 318 318 F F F 18 360-379 188 188 188 315 315 315 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 190 190 190 318 318 318 F F F 21 420-439 186 186 186 312 312 312 F F F 22 440-459 190 190 190 318 318 318 F F F 23 460-479 189 189 189 316 316 316 F F F 24 480-499 192 192 192 321 321 321 F F F 25 500-519 192 192 192 321 321 321 F F F 26 520-539 192 192 192 321 321 321 F F F 27 540-559 191 191 191 320 320 320 F F F 28 560-579 193 193 193 322 322 322 F F F 29 580-599 186 186 186 313 313 313 F F F Max gen: 193, Max flush: 322 Begin loop 97 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:35.978309798Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.02338643Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.024241555Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.024298761Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.024319159Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.024348126Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982} {"msg":"Crucible 95618215-7ec1-428e-a7ce-b251462a9683 has session id: 38578c78-b6ae-4538-991c-628cdc37e224","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.024884719Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Upstairs opts: Upstairs UUID: 95618215-7ec1-428e-a7ce-b251462a9683, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.024937005Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Crucible stats registered with UUID: 95618215-7ec1-428e-a7ce-b251462a9683","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.024966812Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:36.02499294Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025238219Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"io task","client":"2","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025279945Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"io task","client":"0","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025300363Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"io task","client":"1","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025437461Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"task":"crutest"} {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025457269Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982} {"msg":"95618215-7ec1-428e-a7ce-b251462a9683 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025526913Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025547012Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"client":"0","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025573699Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"client":"1","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025606007Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"client":"2","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025640144Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"io task","client":"2","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025712207Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"io task","client":"0","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025751874Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"io task","client":"1","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"ds_connection connected from Ok(127.0.0.1:33566)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025788241Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"io task","client":"2","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"ds_connection connected from Ok(127.0.0.1:41899)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025821268Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"io task","client":"1","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"ds_connection connected from Ok(127.0.0.1:33740)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.025855615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"io task","client":"0","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.027003245Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"client":"2","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.027146192Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"client":"0","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.027591814Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.027660557Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.028220629Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"client":"1","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"[0]R flush_numbers[0..12]: [321, 321, 314, 320, 321, 319, 316, 318, 318, 313, 322, 318]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.028745863Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"[0]R generation[0..12]: [192, 192, 187, 191, 192, 190, 189, 190, 190, 186, 193, 190]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.028790449Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.028826236Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"[1]R flush_numbers[0..12]: [321, 321, 314, 320, 321, 319, 316, 318, 318, 313, 322, 318]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.028860883Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"[1]R generation[0..12]: [192, 192, 187, 191, 192, 190, 189, 190, 190, 186, 193, 190]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.02889392Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.028933466Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"[2]R flush_numbers[0..12]: [321, 321, 314, 320, 321, 319, 316, 318, 318, 313, 322, 318]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.028972593Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"[2]R generation[0..12]: [192, 192, 187, 191, 192, 190, 189, 190, 190, 186, 193, 190]","v":Wait for a query_work_queue command to finish before sending IO 0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029017319Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029055736Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Max found gen is 194","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029088633Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.02912244Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Next flush: 323","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029154777Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029190554Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"":"downstairs","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.02923218Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029249739Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"client":"0","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029267017Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"client":"1","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029284116Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"client":"2","":"downstairs","session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029301554Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029318333Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"95618215-7ec1-428e-a7ce-b251462a9683 is now active with session: 38578c78-b6ae-4538-991c-628cdc37e224","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029336382Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.02935575Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029369488Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982,"session_id":"38578c78-b6ae-4538-991c-628cdc37e224"} {"msg":"Activated sub_volume 95618215-7ec1-428e-a7ce-b251462a9683","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:36.029385907Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 483 len 16384 data: 18 17 15 17 03/30 Flush 04/30 Flush 05/30 Write block 434 len 36864 data: 17 17 17 17 17 17 17 18 18 06/30 Read block 226 len 4096 07/30 Write block 358 len 4096 data: 15 08/30 Flush 09/30 Write block 347 len 40960 data: 19 20 20 19 15 17 18 18 18 14 10/30 Flush 11/30 Write block 588 len 12288 data: 12 13 15 12/30 Read block 542 len 36864 13/30 Write block 425 len 24576 data: 13 15 18 16 17 18 14/30 Write block 219 len 36864 data: 20 19 19 17 13 15 11 10 11 15/30 Write block 364 len 36864 data: 18 19 19 17 14 17 12 10 11 16/30 Read block 78 len 40960 17/30 Read block 47 len 20480 18/30 Flush 19/30 Write block 521 len 36864 data: 12 14 12 11 10 12 13 13 14 20/30 Read block 419 len 24576 21/30 Flush 22/30 Write block 443 len 28672 data: 19 20 19 18 20 18 19 23/30 Flush 24/30 Write block 371 len 12288 data: 11 12 11 25/30 Read block 281 len 8192 26/30 Read block 478 len 28672 27/30 Read block 291 len 36864 28/30 Read block 177 len 40960 29/30 Read block 153 len 28672 30/30 Read block 31 len 16384 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Done Done Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 9 Done Sent Done false 1005 Acked Read 1 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 10 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Read 5 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 1 29 Done 29 1 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1000)) Some(JobId(1022)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:36.05012313Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1982} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 321 321 321 F F F 1 020-039 192 192 192 321 321 321 F F F 2 040-059 187 187 187 314 314 314 F F F 3 060-079 191 191 191 320 320 320 F F F 4 080-099 192 192 192 321 321 321 F F F 5 100-119 190 190 190 319 319 319 F F F 6 120-139 189 189 189 316 316 316 F F F 7 140-159 190 190 190 318 318 318 F F F 8 160-179 190 190 190 318 318 318 F F F 9 180-199 186 186 186 313 313 313 F F F 10 200-219 194 193 194 328 322 328 F F F <--- 11 220-239 194 190 194 328 318 328 F F F <--- 12 240-259 178 178 178 296 296 296 F F F 13 260-279 193 193 193 322 322 322 F F F 14 280-299 192 192 192 321 321 321 F F F 15 300-319 190 190 190 319 319 319 F F F 16 320-339 191 191 191 320 320 320 F F F 17 340-359 194 190 194 327 318 327 F F F <--- 18 360-379 194 188 194 328 315 328 T F T <--- 19 380-399 188 188 188 315 315 315 F F F 20 400-419 190 190 190 318 318 318 F F F 21 420-439 194 186 194 328 312 328 F F F <--- 22 440-459 194 190 194 330 318 330 F F F <--- 23 460-479 189 189 189 316 316 316 F F F 24 480-499 194 192 194 324 321 324 F T F <--- 25 500-519 192 192 192 321 321 321 F F F 26 520-539 194 192 194 329 321 329 F F F <--- 27 540-559 191 191 191 320 320 320 F F F 28 560-579 193 193 193 322 322 322 F F F 29 580-599 194 186 194 328 313 328 F F F <--- Max gen: 194, Max flush: 330 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 321 321 321 F F F 1 020-039 192 192 192 321 321 321 F F F 2 040-059 187 187 187 314 314 314 F F F 3 060-079 191 191 191 320 320 320 F F F 4 080-099 192 192 192 321 321 321 F F F 5 100-119 190 190 190 319 319 319 F F F 6 120-139 189 189 189 316 316 316 F F F 7 140-159 190 190 190 318 318 318 F F F 8 160-179 190 190 190 318 318 318 F F F 9 180-199 186 186 186 313 313 313 F F F 10 200-219 194 194 194 328 328 328 F F F 11 220-239 194 194 194 328 328 328 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 193 193 193 322 322 322 F F F 14 280-299 192 192 192 321 321 321 F F F 15 300-319 190 190 190 319 319 319 F F F 16 320-339 191 191 191 320 320 320 F F F 17 340-359 194 194 194 327 327 327 F F F 18 360-379 195 195 195 331 331 331 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 190 190 190 318 318 318 F F F 21 420-439 194 194 194 328 328 328 F F F 22 440-459 194 194 194 330 330 330 F F F 23 460-479 189 189 189 316 316 316 F F F 24 480-499 194 194 194 324 324 324 F F F 25 500-519 192 192 192 321 321 321 F F F 26 520-539 194 194 194 329 329 329 F F F 27 540-559 191 191 191 320 320 320 F F F 28 560-579 193 193 193 322 322 322 F F F 29 580-599 194 194 194 328 328 328 F F F Max gen: 195, Max flush: 331 Begin loop 98 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.441244183Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.485964756Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.486815812Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.486861788Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.486878226Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.486899995Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990} {"msg":"Crucible d48ab1e5-5f7e-4a84-946f-fc916e417a19 has session id: 6f92df7a-df7c-45f4-8023-517378045d82","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.487489113Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Upstairs opts: Upstairs UUID: d48ab1e5-5f7e-4a84-946f-fc916e417a19, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.487547258Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Crucible stats registered with UUID: d48ab1e5-5f7e-4a84-946f-fc916e417a19","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.487577735Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:40.487604393Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.487790737Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"io task","client":"1","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.487842992Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"io task","client":"0","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.487877869Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"io task","client":"2","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.487992779Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"task":"crutest"} {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488029076Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990} {"msg":"d48ab1e5-5f7e-4a84-946f-fc916e417a19 active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488091621Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488113259Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"client":"0","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488131247Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"client":"1","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488148136Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"client":"2","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488166894Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"io task","client":"2","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488195631Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"io task","client":"0","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488226249Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"io task","client":"1","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"ds_connection connected from Ok(127.0.0.1:59019)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488455899Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"io task","client":"1","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"ds_connection connected from Ok(127.0.0.1:62434)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488503405Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"io task","client":"2","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"ds_connection connected from Ok(127.0.0.1:52249)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.488528122Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"io task","client":"0","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.489741397Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"client":"2","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.489791912Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"client":"0","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490207646Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"client":"1","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490374782Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490413038Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"[0]R flush_numbers[0..12]: [321, 321, 314, 320, 321, 319, 316, 318, 318, 313, 328, 328]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490718952Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"[0]R generation[0..12]: [192, 192, 187, 191, 192, 190, 189, 190, 190, 186, 194, 194]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.49074448Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490765308Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"[1]R flush_numbers[0..12]: [321, 321, 314, 320, 321, 319, 316, 318, 318, 313, 328, 328]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490784476Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"[1]R generation[0..12]: [192, 192, 187, 191, 192, 190, 189, 190, 190, 186, 194, 194]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490803564Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490823542Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"[2]R flush_numbers[0..12]: [321, 321, 314, 320, 321, 319, 316, 318, 318, 313, 328, 328]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490842821Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"[2]R generation[0..12]: [192, 192, 187, 191, 192, 190, 189, 190, 190, 186, 194, 194]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490861989Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490881648Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490897946Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Generation requested: 196 >= found:196","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490915205Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Next flush: 332","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490931683Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490951931Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"":"downstairs","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.49096929Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.490984968Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"client":"0","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.491002067Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"client":"1","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Wait for a query_work_queue command to finish before sending IO Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.491026305Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"client":"2","":"downstairs","session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.491044003Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.491060352Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"d48ab1e5-5f7e-4a84-946f-fc916e417a19 is now active with session: 6f92df7a-df7c-45f4-8023-517378045d82","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.49107837Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.491093669Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990,"session_id":"6f92df7a-df7c-45f4-8023-517378045d82"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.491109888Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990} {"msg":"Activated sub_volume d48ab1e5-5f7e-4a84-946f-fc916e417a19","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:40.491123806Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 26 len 24576 data: 14 13 11 13 13 16 02/30 Write block 523 len 16384 data: 13 12 11 13 03/30 Read block 569 len 12288 04/30 Read block 249 len 32768 05/30 Read block 147 len 8192 06/30 Write block 412 len 16384 data: 10 11 13 14 07/30 Write block 487 len 36864 data: 15 10 10 9 15 18 16 18 18 08/30 Write block 287 len 4096 data: 12 09/30 Read block 480 len 16384 10/30 Write block 566 len 12288 data: 14 14 15 11/30 Read block 342 len 36864 12/30 Read block 280 len 24576 13/30 Read block 29 len 36864 14/30 Write block 63 len 20480 data: 11 13 14 16 18 15/30 Write block 297 len 12288 data: 12 12 18 16/30 Read block 164 len 12288 17/30 Flush 18/30 Flush 19/30 Write block 483 len 12288 data: 19 18 16 20/30 Read block 414 len 40960 21/30 Read block 116 len 40960 22/30 Read block 158 len 28672 23/30 Read block 321 len 12288 24/30 Write block 155 len 36864 data: 13 15 14 12 15 19 23 18 16 25/30 Read block 257 len 40960 26/30 Read block 481 len 4096 27/30 Write block 196 len 16384 data: 18 18 22 21 28/30 Read block 584 len 16384 29/30 Read block 192 len 16384 30/30 Write block 562 len 20480 data: 9 12 14 12 15 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Read 2 Done Done Done false 1005 Acked Write 4 Done Done Done false 1006 Acked Write 9 Done Sent Done false 1007 Acked Write 1 Done Sent Done false 1008 Acked Read 4 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Read 9 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Read 3 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Read 10 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Read 10 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 24 1 26 Done 29 6 29 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) None Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:40.510662834Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1990} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 321 321 321 F F F 1 020-039 196 192 196 332 321 332 F T F <--- 2 040-059 187 187 187 314 314 314 F F F 3 060-079 196 191 196 332 320 332 F F F <--- 4 080-099 192 192 192 321 321 321 F F F 5 100-119 190 190 190 319 319 319 F F F 6 120-139 189 189 189 316 316 316 F F F 7 140-159 190 190 190 318 318 318 T F T <--- 8 160-179 190 190 190 318 318 318 T F T <--- 9 180-199 186 186 186 313 313 313 T F T <--- 10 200-219 194 194 194 328 328 328 F F F 11 220-239 194 194 194 328 328 328 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 193 193 193 322 322 322 F F F 14 280-299 196 192 196 332 321 332 F F F <--- 15 300-319 190 190 190 319 319 319 F F F 16 320-339 191 191 191 320 320 320 F F F 17 340-359 194 194 194 327 327 327 F F F 18 360-379 195 195 195 331 331 331 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 196 190 196 332 318 332 F T F <--- 21 420-439 194 194 194 328 328 328 F F F 22 440-459 194 194 194 330 330 330 F F F 23 460-479 189 189 189 316 316 316 F F F 24 480-499 196 194 196 332 324 332 T F T <--- 25 500-519 192 192 192 321 321 321 F F F 26 520-539 196 194 196 332 329 332 F T F <--- 27 540-559 191 191 191 320 320 320 F F F 28 560-579 196 193 196 332 322 332 T F T <--- 29 580-599 194 194 194 328 328 328 F F F Max gen: 196, Max flush: 332 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 321 321 321 F F F 1 020-039 196 196 196 332 332 332 F F F 2 040-059 187 187 187 314 314 314 F F F 3 060-079 196 196 196 332 332 332 F F F 4 080-099 192 192 192 321 321 321 F F F 5 100-119 190 190 190 319 319 319 F F F 6 120-139 189 189 189 316 316 316 F F F 7 140-159 197 197 197 333 333 333 F F F 8 160-179 197 197 197 333 333 333 F F F 9 180-199 197 197 197 333 333 333 F F F 10 200-219 194 194 194 328 328 328 F F F 11 220-239 194 194 194 328 328 328 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 193 193 193 322 322 322 F F F 14 280-299 196 196 196 332 332 332 F F F 15 300-319 190 190 190 319 319 319 F F F 16 320-339 191 191 191 320 320 320 F F F 17 340-359 194 194 194 327 327 327 F F F 18 360-379 195 195 195 331 331 331 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 196 196 196 332 332 332 F F F 21 420-439 194 194 194 328 328 328 F F F 22 440-459 194 194 194 330 330 330 F F F 23 460-479 189 189 189 316 316 316 F F F 24 480-499 197 197 197 333 333 333 F F F 25 500-519 192 192 192 321 321 321 F F F 26 520-539 196 196 196 332 332 332 F F F 27 540-559 191 191 191 320 320 320 F F F 28 560-579 197 197 197 333 333 333 F F F 29 580-599 194 194 194 328 328 328 F F F Max gen: 197, Max flush: 333 Begin loop 99 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.905871211Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.950417869Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.951192822Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"4c4b956995d7a3110f080eebc9768bd86623f0f1\",\n git_commit_timestamp: \"2025-12-23T23:54:01.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.951240567Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.951257316Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.951279404Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998} {"msg":"Crucible ad652df1-8479-4b74-8f3d-169a4610e3aa has session id: d52bf2b3-4f29-4f37-81fe-31bb1854bb8d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.951692538Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Upstairs opts: Upstairs UUID: ad652df1-8479-4b74-8f3d-169a4610e3aa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.951749483Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Crucible stats registered with UUID: ad652df1-8479-4b74-8f3d-169a4610e3aa","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.951779381Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:44.951806148Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952045108Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"io task","client":"2","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952094123Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"io task","client":"0","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.95212391Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"io task","client":"1","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952262569Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"task":"crutest"} {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952295216Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998} {"msg":"ad652df1-8479-4b74-8f3d-169a4610e3aa active request set","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952348771Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952379768Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"client":"0","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952399917Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"client":"1","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952419515Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"client":"2","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952451892Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"io task","client":"2","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952480859Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"io task","client":"0","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952521196Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"io task","client":"1","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:46019)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952556373Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"io task","client":"2","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:40831)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.95258942Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"io task","client":"0","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:50615)","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.952627047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"io task","client":"1","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.953973499Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"client":"0","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.954022615Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"client":"2","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.954617693Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.954691057Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.954952734Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"client":"1","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"[0]R flush_numbers[0..12]: [321, 332, 314, 332, 321, 319, 316, 333, 333, 333, 328, 328]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955448501Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"[0]R generation[0..12]: [192, 196, 187, 196, 192, 190, 189, 197, 197, 197, 194, 194]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955476018Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955496487Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"[1]R flush_numbers[0..12]: [321, 332, 314, 332, 321, 319, 316, 333, 333, 333, 328, 328]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955515725Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"[1]R generation[0..12]: [192, 196, 187, 196, 192, 190, 189, 197, 197, 197, 194, 194]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955534973Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955557901Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"[2]R flush_numbers[0..12]: [321, 332, 314, 332, 321, 319, 316, 333, 333, 333, 328, 328]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.95557746Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"[2]R generation[0..12]: [192, 196, 187, 196, 192, 190, 189, 197, 197, 197, 194, 194]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955596568Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955620496Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955658802Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Generation requested: 198 >= found:198","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.95568933Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Next flush: 334","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955720227Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955750245Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"":"downstairs","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"No downstairs reconciliation required"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955785151Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955812459Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"client":"0","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955831818Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"client":"1","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955849026Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"client":"2","":"downstairs","session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955866155Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955882393Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"ad652df1-8479-4b74-8f3d-169a4610e3aa is now active with session: d52bf2b3-4f29-4f37-81fe-31bb1854bb8d","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955900312Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.95591839Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998,"session_id":"d52bf2b3-4f29-4f37-81fe-31bb1854bb8d"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955934969Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998} {"msg":"Activated sub_volume ad652df1-8479-4b74-8f3d-169a4610e3aa","v":0,"name":"crucible","level":30,"time":"2025-12-24T00:24:44.955949047Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 334 len 24576 data: 13 12 11 13 10 12 02/30 Write block 32 len 4096 data: 12 03/30 Read block 70 len 8192 04/30 Read block 241 len 8192 05/30 Write block 580 len 36864 data: 12 10 10 11 12 10 8 12 13 06/30 Write block 190 len 32768 data: 14 14 15 16 14 15 19 19 07/30 Read block 278 len 8192 08/30 Write block 322 len 20480 data: 14 15 15 14 13 09/30 Read block 474 len 40960 10/30 Read block 229 len 28672 11/30 Read block 195 len 20480 12/30 Write block 463 len 28672 data: 18 18 15 14 15 13 14 13/30 Read block 456 len 32768 14/30 Read block 81 len 4096 15/30 Read block 533 len 24576 16/30 Read block 269 len 24576 17/30 Write block 127 len 28672 data: 14 14 14 12 14 12 12 18/30 Read block 82 len 8192 19/30 Write block 103 len 12288 data: 14 10 15 20/30 Write block 211 len 28672 data: 15 15 15 16 15 14 18 21/30 Read block 500 len 36864 22/30 Write block 16 len 24576 data: 15 16 16 17 13 14 23/30 Flush 24/30 Write block 108 len 32768 data: 16 17 18 17 15 18 16 14 25/30 Read block 354 len 24576 26/30 Read block 244 len 4096 27/30 Write block 135 len 16384 data: 14 13 13 16 28/30 Write block 46 len 28672 data: 15 13 11 11 9 9 6 29/30 Write block 8 len 4096 data: 14 30/30 Read block 119 len 24576 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Read 2 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Write 8 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Write 1 Done Sent Done false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 1 26 Done 30 5 29 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-12-24T00:24:44.975304041Z","hostname":"w-01KD6V8YXJVY0FPM02DJG3JCWN","pid":1998} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 198 192 198 334 321 334 T F T <--- 1 020-039 198 196 198 334 332 334 F T F <--- 2 040-059 187 187 187 314 314 314 T F T <--- 3 060-079 196 196 196 332 332 332 F F F 4 080-099 192 192 192 321 321 321 F F F 5 100-119 198 190 198 334 319 334 T F T <--- 6 120-139 198 189 198 334 316 334 T F T <--- 7 140-159 197 197 197 333 333 333 F F F 8 160-179 197 197 197 333 333 333 F F F 9 180-199 198 197 198 334 333 334 F F F <--- 10 200-219 198 194 198 334 328 334 F F F <--- 11 220-239 194 194 194 328 328 328 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 193 193 193 322 322 322 F F F 14 280-299 196 196 196 332 332 332 F F F 15 300-319 190 190 190 319 319 319 F F F 16 320-339 198 191 198 334 320 334 F T F <--- 17 340-359 194 194 194 327 327 327 F F F 18 360-379 195 195 195 331 331 331 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 196 196 196 332 332 332 F F F 21 420-439 194 194 194 328 328 328 F F F 22 440-459 194 194 194 330 330 330 F F F 23 460-479 198 189 198 334 316 334 F F F <--- 24 480-499 197 197 197 333 333 333 F F F 25 500-519 192 192 192 321 321 321 F F F 26 520-539 196 196 196 332 332 332 F F F 27 540-559 191 191 191 320 320 320 F F F 28 560-579 197 197 197 333 333 333 F F F 29 580-599 198 194 198 334 328 334 F T F <--- Max gen: 198, Max flush: 334 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 199 199 199 335 335 335 F F F 1 020-039 198 198 198 334 334 334 F F F 2 040-059 199 199 199 335 335 335 F F F 3 060-079 196 196 196 332 332 332 F F F 4 080-099 192 192 192 321 321 321 F F F 5 100-119 199 199 199 335 335 335 F F F 6 120-139 199 199 199 335 335 335 F F F 7 140-159 197 197 197 333 333 333 F F F 8 160-179 197 197 197 333 333 333 F F F 9 180-199 198 198 198 334 334 334 F F F 10 200-219 198 198 198 334 334 334 F F F 11 220-239 194 194 194 328 328 328 F F F 12 240-259 178 178 178 296 296 296 F F F 13 260-279 193 193 193 322 322 322 F F F 14 280-299 196 196 196 332 332 332 F F F 15 300-319 190 190 190 319 319 319 F F F 16 320-339 198 198 198 334 334 334 F F F 17 340-359 194 194 194 327 327 327 F F F 18 360-379 195 195 195 331 331 331 F F F 19 380-399 188 188 188 315 315 315 F F F 20 400-419 196 196 196 332 332 332 F F F 21 420-439 194 194 194 328 328 328 F F F 22 440-459 194 194 194 330 330 330 F F F 23 460-479 198 198 198 334 334 334 F F F 24 480-499 197 197 197 333 333 333 F F F 25 500-519 192 192 192 321 321 321 F F F 26 520-539 196 196 196 332 332 332 F F F 27 540-559 191 191 191 320 320 320 F F F 28 560-579 197 197 197 333 333 333 F F F 29 580-599 198 198 198 334 334 334 F F F Max gen: 199, Max flush: 335 9:41 Test duration Test completed real 9:40.587993326 user 58.595926194 sys 25.190925015 trap 0.081956805 tflt 0.088797583 dflt 0.210583186 kflt 0.006432003 lock 15:08:56.224346789 slp 46:32.916070628 lat 6.353065356 stop 0.033603178