Turn off color for downstairs dump Removing existing region "/var/tmp/test_repair" Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/test_live_repair/test_repair_dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.20209602 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.013195052 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.012837863 Created 3 regions /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1135 1136 1137 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.715364957Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.731032117Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.733309924Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.733469331Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.733484459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.733503516Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140} {"msg":"Crucible b979eb5e-b55e-4e45-a495-9630c5b181c2 has session id: 67512375-3220-4e89-b4e0-86da5f882b16","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.734375149Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Upstairs opts: Upstairs UUID: b979eb5e-b55e-4e45-a495-9630c5b181c2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.734532566Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Crucible stats registered with UUID: b979eb5e-b55e-4e45-a495-9630c5b181c2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.734551003Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:23.734602025Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735165693Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"io task","client":"2","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735415477Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"io task","client":"0","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735429814Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"io task","client":"1","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735522501Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735690756Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140} {"msg":"b979eb5e-b55e-4e45-a495-9630c5b181c2 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735748458Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735757857Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"client":"0","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735893077Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"client":"1","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735903335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"client":"2","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.73594082Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"io task","client":"2","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735951818Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"io task","client":"0","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.735961837Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"io task","client":"1","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"ds_connection connected from Ok(127.0.0.1:58555)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.736462204Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"io task","client":"0","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"ds_connection connected from Ok(127.0.0.1:35608)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.73648748Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"io task","client":"1","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"ds_connection connected from Ok(127.0.0.1:56895)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.736498998Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"io task","client":"2","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.738364226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"client":"1","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.738391872Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"client":"0","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.738451853Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"client":"2","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739060804Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739074962Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739273503Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739284351Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739295379Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739305628Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739315796Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739326525Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739336553Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739346492Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.73935729Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739366759Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739376418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739385666Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739396315Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"":"downstairs","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739406173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739414862Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"client":"0","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739424881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"client":"1","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739434699Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"client":"2","":"downstairs","session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739444668Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739453676Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"b979eb5e-b55e-4e45-a495-9630c5b181c2 is now active with session: 67512375-3220-4e89-b4e0-86da5f882b16","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739463465Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739472404Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.739644348Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140,"session_id":"67512375-3220-4e89-b4e0-86da5f882b16"} {"msg":"Activated sub_volume b979eb5e-b55e-4e45-a495-9630c5b181c2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:23.740125898Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1140} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Fill test Read and Verify all blocks (0..600 range:false) Wrote out file "/tmp/test_live_repair/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Read 88 Done Done Done false 1005 Acked Read 256 Done Done Done false 1006 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1006 1005 1004 1003 1002 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1007 1006 1005 1004 1003 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program Begin loop 1 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.805475738Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.813779515Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814076391Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814103517Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814111936Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814123345Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142} {"msg":"Crucible be610e31-926c-46a2-a8a4-2fd9c38918a1 has session id: 909e0a73-49ab-4ffc-8257-883a6f4b66fd","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814446257Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Upstairs opts: Upstairs UUID: be610e31-926c-46a2-a8a4-2fd9c38918a1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814491301Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Crucible stats registered with UUID: be610e31-926c-46a2-a8a4-2fd9c38918a1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814501859Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:27.814510938Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814665195Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"io task","client":"0","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.81470289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"io task","client":"2","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814714208Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"io task","client":"1","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.81483835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"task":"crutest"} {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814858677Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142} {"msg":"be610e31-926c-46a2-a8a4-2fd9c38918a1 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814889103Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814898531Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"client":"0","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.81490817Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"client":"1","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814917748Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"client":"2","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814927207Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"io task","client":"2","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.814937006Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"io task","client":"0","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.81497555Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"io task","client":"1","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"ds_connection connected from Ok(127.0.0.1:42967)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.815002516Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"io task","client":"0","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"ds_connection connected from Ok(127.0.0.1:53490)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.815074815Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"io task","client":"1","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"ds_connection connected from Ok(127.0.0.1:34328)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.815104261Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"io task","client":"2","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.815646192Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"client":"0","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.815667939Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"client":"1","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.815880588Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"client":"2","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816026016Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816050603Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816166056Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816177484Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816188493Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816198701Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.81620869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816219728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816229887Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816239955Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816250714Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816260142Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816269651Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816278879Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816288228Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"":"downstairs","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816297797Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816307805Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"client":"0","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816317514Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"client":"1","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816326832Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"Wait for a query_work_queue command to finish before sending IO client":"2","":"downstairs","session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.81634032Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816349409Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"be610e31-926c-46a2-a8a4-2fd9c38918a1 is now active with session: 909e0a73-49ab-4ffc-8257-883a6f4b66fd","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816359218Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816368236Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816375605Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142,"session_id":"909e0a73-49ab-4ffc-8257-883a6f4b66fd"} {"msg":"Activated sub_volume be610e31-926c-46a2-a8a4-2fd9c38918a1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:27.816384634Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 417 len 24576 data: 2 2 2 2 2 2 02/30 Read block 252 len 12288 03/30 Write block 139 len 16384 data: 2 2 2 2 04/30 Write block 456 len 32768 data: 2 2 2 2 2 2 2 2 05/30 Read block 134 len 24576 06/30 Write block 261 len 36864 data: 2 2 2 2 2 2 2 2 2 07/30 Read block 200 len 12288 08/30 Read block 280 len 8192 09/30 Write block 441 len 32768 data: 2 2 2 2 2 2 2 2 10/30 Write block 156 len 40960 data: 2 2 2 2 2 2 2 2 2 2 11/30 Read block 9 len 12288 12/30 Write block 514 len 16384 data: 2 2 2 2 13/30 Read block 486 len 28672 14/30 Read block 358 len 28672 15/30 Flush 16/30 Write block 81 len 20480 data: 2 2 2 2 2 17/30 Write block 529 len 36864 data: 2 2 2 2 2 2 2 2 2 18/30 Write block 410 len 40960 data: 2 2 2 2 2 2 2 3 3 3 19/30 Read block 161 len 20480 20/30 Write block 389 len 20480 data: 2 2 2 2 2 21/30 Read block 26 len 12288 22/30 Read block 169 len 40960 23/30 Read block 338 len 36864 24/30 Write block 107 len 40960 data: 2 2 2 2 2 2 2 2 2 2 25/30 Read block 299 len 20480 26/30 Write block 574 len 12288 data: 2 2 2 27/30 Write block 303 len 16384 data: 2 2 2 2 28/30 Write block 441 len 36864 data: 3 3 3 3 3 3 3 3 2 29/30 Read block 25 len 8192 30/30 Write block 317 len 32768 data: 2 2 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Sent false 1001 Acked Read 3 Done Done Sent false 1002 Acked Write 4 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Read 6 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Read 3 Done Done Sent false 1007 Acked Read 2 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Read 3 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Write 5 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Read 5 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 2 Done Done Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1014)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:27.826339939Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1142} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 1 1 1 1 1 1 T T F <--- 5 100-119 1 1 1 1 1 1 T T F <--- 6 120-139 2 2 1 2 2 1 F F F <--- 7 140-159 2 2 1 2 2 1 F F F <--- 8 160-179 2 2 1 2 2 1 F F F <--- 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 2 2 1 2 2 1 F F F <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 T T F <--- 16 320-339 1 1 1 1 1 1 T T F <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 1 1 1 1 1 1 F F F 19 380-399 1 1 1 1 1 1 T T F <--- 20 400-419 2 2 1 2 2 1 T T F <--- 21 420-439 2 2 1 2 2 1 F F F <--- 22 440-459 2 2 1 2 2 1 T T F <--- 23 460-479 2 2 1 2 2 1 F F F <--- 24 480-499 1 1 1 1 1 1 F F F 25 500-519 2 2 1 2 2 1 F F F <--- 26 520-539 1 1 1 1 1 1 T T F <--- 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 T T F <--- 29 580-599 1 1 1 1 1 1 F F F Max gen: 2, Max flush: 2 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 3 3 3 3 3 3 F F F 5 100-119 3 3 3 3 3 3 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 2 2 2 2 2 2 F F F 8 160-179 2 2 2 2 2 2 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 2 2 2 2 2 2 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 3 3 3 3 3 3 F F F 16 320-339 3 3 3 3 3 3 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 1 1 1 1 1 1 F F F 19 380-399 3 3 3 3 3 3 F F F 20 400-419 3 3 3 3 3 3 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 2 2 2 2 2 2 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 2 2 2 2 2 2 F F F 26 520-539 3 3 3 3 3 3 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 3 3 3 3 3 3 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 3, Max flush: 3 Begin loop 2 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.217998315Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.226215094Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.226501832Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.226528398Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.226536887Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.226548245Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150} {"msg":"Crucible 577252dd-61d7-4e43-987f-de2156ff196d has session id: 08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.226843692Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Upstairs opts: Upstairs UUID: 577252dd-61d7-4e43-987f-de2156ff196d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.226874438Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Crucible stats registered with UUID: 577252dd-61d7-4e43-987f-de2156ff196d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.226886776Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:32.226896704Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227039294Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"io task","client":"1","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.22706264Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"io task","client":"0","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227073619Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"io task","client":"2","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227246983Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"task":"crutest"} {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.22727214Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150} {"msg":"577252dd-61d7-4e43-987f-de2156ff196d active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227297556Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227307424Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"client":"0","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227317643Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"client":"1","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227327651Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"client":"2","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.22733766Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"io task","client":"2","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227348218Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"io task","client":"1","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227397021Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"io task","client":"0","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:38534)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227424377Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"io task","client":"2","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:38081)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.22754497Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"io task","client":"1","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:48989)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.227560707Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"io task","client":"0","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228098949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"client":"2","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228176887Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"client":"0","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228218571Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"client":"1","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228388156Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228484692Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 3, 3, 2, 2, 2, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228541604Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 3, 3, 2, 2, 2, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228552262Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228563071Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 3, 3, 2, 2, 2, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228573159Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 3, 3, 2, 2, 2, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228583028Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228593696Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 3, 3, 2, 2, 2, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228606465Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 3, 3, 2, 2, 2, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228616383Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228627092Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.22863633Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228646219Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Next flush: 4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228655497Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228664716Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"":"downstairs","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228674195Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228682943Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"client":"0","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228692372Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"client":"1","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:10:32.228701781Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"client":"2","":"downstairs","session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228714249Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228723147Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"577252dd-61d7-4e43-987f-de2156ff196d is now active with session: 08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228733026Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228741975Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228749544Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150,"session_id":"08b68c59-9e44-4bcd-8b0c-8f4e5f4c41ab"} {"msg":"Activated sub_volume 577252dd-61d7-4e43-987f-de2156ff196d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:32.228759832Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 585 len 16384 data: 2 2 2 2 02/30 Flush 03/30 Write block 473 len 8192 data: 2 2 04/30 Write block 575 len 20480 data: 3 3 2 2 2 05/30 Read block 414 len 12288 06/30 Read block 59 len 28672 07/30 Write block 26 len 8192 data: 2 2 08/30 Read block 263 len 8192 09/30 Write block 360 len 8192 data: 2 2 10/30 Read block 197 len 40960 11/30 Flush 12/30 Write block 94 len 16384 data: 2 2 2 2 13/30 Flush 14/30 Write block 25 len 24576 data: 2 3 3 2 2 2 15/30 Read block 529 len 20480 16/30 Write block 560 len 32768 data: 2 2 2 2 2 2 2 2 17/30 Read block 137 len 8192 18/30 Read block 15 len 36864 19/30 Read block 183 len 20480 20/30 Write block 96 len 36864 data: 3 3 2 2 2 2 2 2 2 21/30 Write block 214 len 28672 data: 2 2 2 2 2 2 2 22/30 Write block 45 len 32768 data: 2 2 2 2 2 2 2 2 23/30 Read block 177 len 40960 24/30 Flush 25/30 Write block 145 len 36864 data: 2 2 2 2 2 2 2 2 2 26/30 Read block 310 len 28672 27/30 Write block 263 len 36864 data: 3 3 3 3 3 3 3 2 2 28/30 Read block 541 len 8192 29/30 Write block 335 len 8192 data: 2 2 30/30 Write block 204 len 12288 data: 2 2 2 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 2 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Read 7 Done Done Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Read 2 Done Sent Done false 1008 Acked Write 2 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 4 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Read 5 Done Sent Done false 1015 Acked Write 8 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Read 10 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 24 2 28 Done 26 4 26 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1001)) Some(JobId(1023)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:32.237627226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1150} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 4 1 4 7 1 7 F F F <--- 2 040-059 4 1 4 7 1 7 F F F <--- 3 060-079 1 1 1 1 1 1 F F F 4 080-099 4 3 4 7 3 7 F F F <--- 5 100-119 4 3 4 7 3 7 F F F <--- 6 120-139 2 2 2 2 2 2 F F F 7 140-159 2 2 2 2 2 2 T F T <--- 8 160-179 2 2 2 2 2 2 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 4 1 4 7 1 7 T F T <--- 11 220-239 4 1 4 7 1 7 F F F <--- 12 240-259 1 1 1 1 1 1 F F F 13 260-279 2 2 2 2 2 2 T F T <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 3 3 3 3 3 3 F F F 16 320-339 3 3 3 3 3 3 T F T <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 4 1 4 5 1 5 F F F <--- 19 380-399 3 3 3 3 3 3 F F F 20 400-419 3 3 3 3 3 3 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 2 4 5 2 5 F T F <--- 24 480-499 1 1 1 1 1 1 F F F 25 500-519 2 2 2 2 2 2 F F F 26 520-539 3 3 3 3 3 3 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 4 3 4 7 3 7 F T F <--- 29 580-599 4 4 4 4 4 4 F F F Max gen: 4, Max flush: 7 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 4 4 4 7 7 7 F F F 2 040-059 4 4 4 7 7 7 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 4 4 4 7 7 7 F F F 5 100-119 4 4 4 7 7 7 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 5 5 5 8 8 8 F F F 8 160-179 2 2 2 2 2 2 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 5 5 5 8 8 8 F F F 11 220-239 4 4 4 7 7 7 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 5 5 5 8 8 8 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 3 3 3 3 3 3 F F F 16 320-339 5 5 5 8 8 8 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 4 4 4 5 5 5 F F F 19 380-399 3 3 3 3 3 3 F F F 20 400-419 3 3 3 3 3 3 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 2 2 2 2 2 2 F F F 26 520-539 3 3 3 3 3 3 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 4 4 4 4 4 4 F F F Max gen: 5, Max flush: 8 Begin loop 3 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.477432598Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:36.48467271Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.484730801Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.491740147Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492014116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492049711Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.49205909Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492071078Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158} {"msg":"Crucible 8e84903a-9158-429c-9518-fac428493c49 has session id: ca27069d-f74d-41d4-a605-2130930839f5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492424007Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Upstairs opts: Upstairs UUID: 8e84903a-9158-429c-9518-fac428493c49, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492453762Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Crucible stats registered with UUID: 8e84903a-9158-429c-9518-fac428493c49","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492465091Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:36.492474369Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492610109Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"io task","client":"0","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492634206Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"io task","client":"1","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492646714Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"io task","client":"2","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492770636Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"task":"crutest"} {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492796232Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158} {"msg":"8e84903a-9158-429c-9518-fac428493c49 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.49281266Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492821628Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"client":"0","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492831417Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"client":"1","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492841086Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"client":"2","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492855014Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"io task","client":"2","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492865162Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"io task","client":"0","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492875231Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"io task","client":"1","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37750)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492911395Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"io task","client":"0","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:41851)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492922314Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"io task","client":"2","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:64907)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.492932562Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"io task","client":"1","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.493607434Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"client":"1","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.493642898Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"client":"0","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.493757202Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"client":"2","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.493912869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.493937235Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"[0]R flush_numbers[0..12]: [1, 7, 7, 1, 7, 7, 2, 8, 2, 1, 8, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494033111Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"[0]R generation[0..12]: [1, 4, 4, 1, 4, 4, 2, 5, 2, 1, 5, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.49404425Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494055308Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"[1]R flush_numbers[0..12]: [1, 7, 7, 1, 7, 7, 2, 8, 2, 1, 8, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494065557Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"[1]R generation[0..12]: [1, 4, 4, 1, 4, 4, 2, 5, 2, 1, 5, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494075585Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494086404Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"[2]R flush_numbers[0..12]: [1, 7, 7, 1, 7, 7, 2, 8, 2, 1, 8, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494096692Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"[2]R generation[0..12]: [1, 4, 4, 1, 4, 4, 2, 5, 2, 1, 5, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494106661Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494117639Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Max found gen is 6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494127028Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Generation requested: 6 >= found:6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494136766Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Next flush: 9","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494145875Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494155214Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"":"downstairs","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494166322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494174901Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"client":"0","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494184829Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"client":"1","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494194738Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"client":"2","":"downstairs","session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494204656Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494213765Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"8e84903a-9158-429c-9518-fac428493c49 is now active with session: ca27069d-f74d-41d4-a605-2130930839f5","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:10:36.494223464Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494235412Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494242841Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158,"session_id":"ca27069d-f74d-41d4-a605-2130930839f5"} {"msg":"Activated sub_volume 8e84903a-9158-429c-9518-fac428493c49","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:36.494251799Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 124 len 16384 data: 2 2 2 2 02/30 Read block 211 len 16384 03/30 Write block 587 len 24576 data: 3 3 2 2 2 2 04/30 Flush 05/30 Write block 80 len 4096 data: 2 06/30 Write block 344 len 24576 data: 2 2 2 2 2 2 07/30 Read block 383 len 28672 08/30 Write block 422 len 32768 data: 3 2 2 2 2 2 2 2 09/30 Write block 340 len 12288 data: 2 2 2 10/30 Read block 364 len 28672 11/30 Write block 495 len 16384 data: 2 2 2 2 12/30 Write block 336 len 32768 data: 3 2 2 2 3 3 3 2 13/30 Write block 85 len 28672 data: 3 2 2 2 2 2 2 14/30 Write block 28 len 8192 data: 3 3 15/30 Read block 11 len 36864 16/30 Read block 325 len 40960 17/30 Read block 104 len 32768 18/30 Read block 392 len 20480 19/30 Flush 20/30 Write block 97 len 20480 data: 4 3 3 3 3 21/30 Write block 528 len 40960 data: 2 3 3 3 3 3 3 3 3 3 22/30 Read block 408 len 8192 23/30 Read block 299 len 12288 24/30 Flush 25/30 Write block 349 len 16384 data: 3 2 2 2 26/30 Flush 27/30 Write block 350 len 40960 data: 3 3 3 2 2 2 2 2 2 2 28/30 Write block 275 len 12288 data: 2 2 2 29/30 Write block 382 len 12288 data: 2 2 2 30/30 Write block 120 len 20480 data: 2 2 2 2 3 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 1 Done Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Write 3 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Write 2 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 10 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 10 Sent Sent Sent false 1027 Acked Write 3 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 4 4 33 Done 1 22 22 45 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:36.502330649Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1158} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 4 6 6 7 10 10 F F F <--- 2 040-059 4 4 4 7 7 7 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 4 6 6 7 11 11 T F F <--- 5 100-119 4 6 6 7 11 11 F F F <--- 6 120-139 6 6 6 9 9 9 F T T <--- 7 140-159 5 5 5 8 8 8 F F F 8 160-179 2 2 2 2 2 2 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 5 5 5 8 8 8 F F F 11 220-239 4 4 4 7 7 7 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 5 5 5 8 8 8 F T T <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 3 3 3 3 3 3 F F F 16 320-339 5 6 6 8 10 10 F F F <--- 17 340-359 1 6 6 1 12 12 F T T <--- 18 360-379 4 4 4 5 5 5 F F F 19 380-399 3 3 3 3 3 3 F T T <--- 20 400-419 3 3 3 3 3 3 F F F 21 420-439 2 6 6 2 10 10 F F F <--- 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 1 6 6 1 10 10 F F F <--- 25 500-519 2 2 2 2 2 2 F F F 26 520-539 3 6 6 3 11 11 F F F <--- 27 540-559 1 1 1 1 1 1 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 6 6 6 9 9 9 F F F Max gen: 6, Max flush: 12 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 4 4 4 7 7 7 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 6 6 6 11 11 11 F F F 6 120-139 7 7 7 13 13 13 F F F 7 140-159 5 5 5 8 8 8 F F F 8 160-179 2 2 2 2 2 2 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 5 5 5 8 8 8 F F F 11 220-239 4 4 4 7 7 7 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 7 7 7 13 13 13 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 3 3 3 3 3 3 F F F 16 320-339 6 6 6 10 10 10 F F F 17 340-359 7 7 7 13 13 13 F F F 18 360-379 4 4 4 5 5 5 F F F 19 380-399 7 7 7 13 13 13 F F F 20 400-419 3 3 3 3 3 3 F F F 21 420-439 6 6 6 10 10 10 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 6 6 6 10 10 10 F F F 25 500-519 2 2 2 2 2 2 F F F 26 520-539 6 6 6 11 11 11 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 6 6 6 9 9 9 F F F Max gen: 7, Max flush: 13 Begin loop 4 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.749669299Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:40.7568463Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.756908341Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.763844927Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764109549Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764137515Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764145943Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764159501Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166} {"msg":"Crucible 49e96645-d146-4b22-b4b9-6d42d3341c78 has session id: fa521827-3c80-4f90-b9df-86621652adac","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.76450819Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Upstairs opts: Upstairs UUID: 49e96645-d146-4b22-b4b9-6d42d3341c78, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764536946Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Crucible stats registered with UUID: 49e96645-d146-4b22-b4b9-6d42d3341c78","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764548025Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:40.764557203Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764698593Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"io task","client":"2","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764735517Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"io task","client":"0","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764746566Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"io task","client":"1","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.76485122Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"task":"crutest"} {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764875447Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166} {"msg":"49e96645-d146-4b22-b4b9-6d42d3341c78 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764889235Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764898133Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"client":"0","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764907692Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"client":"1","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764917301Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"client":"2","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764926849Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"io task","client":"2","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764938397Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"io task","client":"0","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764948226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"io task","client":"1","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"ds_connection connected from Ok(127.0.0.1:33784)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.764959344Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"io task","client":"2","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"ds_connection connected from Ok(127.0.0.1:38410)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.765003398Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"io task","client":"0","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"ds_connection connected from Ok(127.0.0.1:61492)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.765074248Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"io task","client":"1","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.765700296Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"client":"2","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.765753978Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"client":"1","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.765769946Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"client":"0","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766000012Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766021869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"[0]R flush_numbers[0..12]: [1, 10, 7, 1, 11, 11, 13, 8, 2, 1, 8, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766089659Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"[0]R generation[0..12]: [1, 6, 4, 1, 6, 6, 7, 5, 2, 1, 5, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766100588Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766111396Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"[1]R flush_numbers[0..12]: [1, 10, 7, 1, 11, 11, 13, 8, 2, 1, 8, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766121565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"[1]R generation[0..12]: [1, 6, 4, 1, 6, 6, 7, 5, 2, 1, 5, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766131513Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766144651Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"[2]R flush_numbers[0..12]: [1, 10, 7, 1, 11, 11, 13, 8, 2, 1, 8, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.76615483Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"[2]R generation[0..12]: [1, 6, 4, 1, 6, 6, 7, 5, 2, 1, 5, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766164928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766175757Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Max found gen is 8","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766185065Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766194594Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Next flush: 14","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766203823Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766213231Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"":"downstairs","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.76622303Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766231688Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"client":"0","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766241207Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"client":"1","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766250736Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"client":"2","":"downstairs","session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766260314Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766269353Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"49e96645-d146-4b22-b4b9-6d42d3341c78 is now active with session: fa521827-3c80-4f90-b9df-86621652adac","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766279152Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166Wait for a query_work_queue command to finish before sending IO ,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766294399Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766301838Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166,"session_id":"fa521827-3c80-4f90-b9df-86621652adac"} {"msg":"Activated sub_volume 49e96645-d146-4b22-b4b9-6d42d3341c78","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:40.766310757Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 533 len 16384 data: 4 4 4 4 02/30 Write block 316 len 4096 data: 2 03/30 Read block 271 len 12288 04/30 Read block 507 len 12288 05/30 Read block 401 len 20480 06/30 Flush 07/30 Write block 313 len 40960 data: 2 2 2 3 3 3 3 3 3 3 08/30 Read block 413 len 32768 09/30 Write block 514 len 4096 data: 3 10/30 Read block 402 len 36864 11/30 Read block 74 len 24576 12/30 Read block 298 len 32768 13/30 Write block 535 len 8192 data: 5 5 14/30 Read block 49 len 12288 15/30 Read block 398 len 20480 16/30 Read block 244 len 36864 17/30 Flush 18/30 Write block 105 len 16384 data: 2 2 3 3 19/30 Write block 105 len 28672 data: 3 3 4 4 3 3 3 20/30 Read block 412 len 12288 21/30 Read block 568 len 40960 22/30 Flush 23/30 Flush 24/30 Write block 544 len 32768 data: 2 2 2 2 2 2 2 2 25/30 Read block 185 len 36864 26/30 Read block 471 len 8192 27/30 Read block 159 len 8192 28/30 Write block 291 len 16384 data: 2 2 2 2 29/30 Write block 134 len 32768 data: 2 2 2 2 2 3 3 3 30/30 Write block 198 len 16384 data: 2 2 2 2 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Sent Done Done false 1001 Acked Write 1 Sent Done Done false 1002 Acked Read 3 Sent Done Done false 1003 Acked Read 3 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 10 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Read 10 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:40.775010405Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1166} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 4 4 4 7 7 7 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 6 8 8 11 16 16 F F F <--- 6 120-139 7 7 7 13 13 13 F T T <--- 7 140-159 5 5 5 8 8 8 F T T <--- 8 160-179 2 2 2 2 2 2 F F F 9 180-199 1 1 1 1 1 1 F T T <--- 10 200-219 5 5 5 8 8 8 F T T <--- 11 220-239 4 4 4 7 7 7 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 7 7 7 13 13 13 F F F 14 280-299 1 1 1 1 1 1 F T T <--- 15 300-319 3 8 8 3 15 15 F F F <--- 16 320-339 6 8 8 10 15 15 F F F <--- 17 340-359 7 7 7 13 13 13 F F F 18 360-379 4 4 4 5 5 5 F F F 19 380-399 7 7 7 13 13 13 F F F 20 400-419 3 3 3 3 3 3 F F F 21 420-439 6 6 6 10 10 10 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 6 6 6 10 10 10 F F F 25 500-519 2 8 8 2 15 15 F F F <--- 26 520-539 6 8 8 11 15 15 F F F <--- 27 540-559 1 1 1 1 1 1 F T T <--- 28 560-579 4 4 4 7 7 7 F F F 29 580-599 6 6 6 9 9 9 F F F Max gen: 8, Max flush: 16 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 4 4 4 7 7 7 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 8 8 8 16 16 16 F F F 6 120-139 9 9 9 17 17 17 F F F 7 140-159 9 9 9 17 17 17 F F F 8 160-179 2 2 2 2 2 2 F F F 9 180-199 9 9 9 17 17 17 F F F 10 200-219 9 9 9 17 17 17 F F F 11 220-239 4 4 4 7 7 7 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 7 7 7 13 13 13 F F F 14 280-299 9 9 9 17 17 17 F F F 15 300-319 8 8 8 15 15 15 F F F 16 320-339 8 8 8 15 15 15 F F F 17 340-359 7 7 7 13 13 13 F F F 18 360-379 4 4 4 5 5 5 F F F 19 380-399 7 7 7 13 13 13 F F F 20 400-419 3 3 3 3 3 3 F F F 21 420-439 6 6 6 10 10 10 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 6 6 6 10 10 10 F F F 25 500-519 8 8 8 15 15 15 F F F 26 520-539 8 8 8 15 15 15 F F F 27 540-559 9 9 9 17 17 17 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 6 6 6 9 9 9 F F F Max gen: 9, Max flush: 17 Begin loop 5 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.025060529Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.033417078Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.033714714Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.03374003Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.033749009Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.033760388Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174} {"msg":"Crucible 3ea58455-ee53-43e3-94a3-0d0cf1b7c90e has session id: 9634f231-5098-449f-8997-18cf7fa3ff8d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034077941Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Upstairs opts: Upstairs UUID: 3ea58455-ee53-43e3-94a3-0d0cf1b7c90e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034107017Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Crucible stats registered with UUID: 3ea58455-ee53-43e3-94a3-0d0cf1b7c90e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034118165Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:45.034127414Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034284821Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"io task","client":"0","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034307418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"io task","client":"1","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034317586Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"io task","client":"2","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034398994Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"task":"crutest"} {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034425Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174} {"msg":"3ea58455-ee53-43e3-94a3-0d0cf1b7c90e active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034440018Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034448867Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"client":"0","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034458605Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"client":"1","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034468154Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"client":"2","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034477673Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"io task","client":"2","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034487471Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"io task","client":"0","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.03456381Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"io task","client":"1","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:42947)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034592046Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"io task","client":"0","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:48134)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034602544Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"io task","client":"2","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:46255)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.034612793Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"io task","client":"1","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.03524945Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"client":"0","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035276166Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"client":"1","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035288064Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"client":"2","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035535438Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035570453Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"[0]R flush_numbers[0..12]: [1, 10, 7, 1, 11, 16, 17, 17, 2, 17, 17, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035616586Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"[0]R generation[0..12]: [1, 6, 4, 1, 6, 8, 9, 9, 2, 9, 9, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035627425Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035638443Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"[1]R flush_numbers[0..12]: [1, 10, 7, 1, 11, 16, 17, 17, 2, 17, 17, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035648622Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"[1]R generation[0..12]: [1, 6, 4, 1, 6, 8, 9, 9, 2, 9, 9, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.03565865Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035669498Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"[2]R flush_numbers[0..12]: [1, 10, 7, 1, 11, 16, 17, 17, 2, 17, 17, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035679787Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"[2]R generation[0..12]: [1, 6, 4, 1, 6, 8, 9, 9, 2, 9, 9, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035689736Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035700604Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035709933Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035719511Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Next flush: 18","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.03572881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035738188Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"":"downstairs","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035749077Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035757756Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"client":"0","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO time":"2026-04-15T18:10:45.035767194Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"client":"1","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035781392Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"client":"2","":"downstairs","session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035791001Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035799959Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"3ea58455-ee53-43e3-94a3-0d0cf1b7c90e is now active with session: 9634f231-5098-449f-8997-18cf7fa3ff8d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035809708Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035818557Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035826136Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174,"session_id":"9634f231-5098-449f-8997-18cf7fa3ff8d"} {"msg":"Activated sub_volume 3ea58455-ee53-43e3-94a3-0d0cf1b7c90e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:45.035835064Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 330 len 20480 data: 2 2 2 2 2 02/30 Read block 182 len 32768 03/30 Read block 51 len 12288 04/30 Write block 523 len 28672 data: 2 2 2 2 2 3 4 05/30 Read block 112 len 20480 06/30 Flush 07/30 Write block 106 len 36864 data: 4 5 5 4 4 4 3 3 3 08/30 Write block 162 len 20480 data: 3 3 3 3 2 09/30 Read block 139 len 20480 10/30 Read block 232 len 4096 11/30 Write block 546 len 40960 data: 3 3 3 3 3 3 2 2 2 2 12/30 Flush 13/30 Write block 586 len 16384 data: 3 4 4 3 14/30 Write block 10 len 40960 data: 2 2 2 2 2 2 2 2 2 2 15/30 Write block 279 len 24576 data: 2 2 2 2 2 2 16/30 Read block 133 len 40960 17/30 Flush 18/30 Write block 140 len 16384 data: 4 4 3 2 19/30 Read block 348 len 32768 20/30 Read block 523 len 32768 21/30 Read block 235 len 20480 22/30 Read block 165 len 12288 23/30 Read block 403 len 32768 24/30 Write block 521 len 40960 data: 2 2 3 3 3 3 3 4 5 4 25/30 Write block 352 len 40960 data: 4 3 3 3 3 3 3 3 3 3 26/30 Read block 273 len 32768 27/30 Write block 115 len 24576 data: 3 3 2 2 2 3 28/30 Write block 133 len 4096 data: 2 29/30 Write block 360 len 8192 data: 4 4 30/30 Write block 406 len 12288 data: 2 2 2 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 7 Done Done Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 9 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Read 5 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 4 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Read 3 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Write 6 Sent Sent Sent false 1027 Acked Write 1 Sent Sent Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 4 26 4 34 Done 26 4 26 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) None Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:45.044725695Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1174} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 1 10 20 1 20 F F F <--- 1 020-039 6 6 6 10 10 10 F F F 2 040-059 4 4 4 7 7 7 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 10 8 10 19 16 19 T F T <--- 6 120-139 9 9 9 17 17 17 T F T <--- 7 140-159 9 9 9 17 17 17 T F T <--- 8 160-179 10 2 10 19 2 19 F F F <--- 9 180-199 9 9 9 17 17 17 F F F 10 200-219 9 9 9 17 17 17 F F F 11 220-239 4 4 4 7 7 7 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 10 7 10 20 13 20 F F F <--- 14 280-299 10 9 10 20 17 20 F F F <--- 15 300-319 8 8 8 15 15 15 F F F 16 320-339 10 8 10 18 15 18 F T F <--- 17 340-359 7 7 7 13 13 13 T F T <--- 18 360-379 4 4 4 5 5 5 T F T <--- 19 380-399 7 7 7 13 13 13 F F F 20 400-419 3 3 3 3 3 3 T F T <--- 21 420-439 6 6 6 10 10 10 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 6 6 6 10 10 10 F F F 25 500-519 8 8 8 15 15 15 F F F 26 520-539 10 8 10 18 15 18 T T T <--- 27 540-559 10 9 10 19 17 19 F F F <--- 28 560-579 4 4 4 7 7 7 F F F 29 580-599 10 6 10 20 9 20 F F F <--- Max gen: 10, Max flush: 20 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 20 20 20 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 4 4 4 7 7 7 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 11 11 11 21 21 21 F F F 7 140-159 11 11 11 21 21 21 F F F 8 160-179 10 10 10 19 19 19 F F F 9 180-199 9 9 9 17 17 17 F F F 10 200-219 9 9 9 17 17 17 F F F 11 220-239 4 4 4 7 7 7 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 10 10 10 20 20 20 F F F 14 280-299 10 10 10 20 20 20 F F F 15 300-319 8 8 8 15 15 15 F F F 16 320-339 10 10 10 18 18 18 F F F 17 340-359 11 11 11 21 21 21 F F F 18 360-379 11 11 11 21 21 21 F F F 19 380-399 7 7 7 13 13 13 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 6 6 6 10 10 10 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 6 6 6 10 10 10 F F F 25 500-519 8 8 8 15 15 15 F F F 26 520-539 11 11 11 21 21 21 F F F 27 540-559 10 10 10 19 19 19 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 10 10 10 20 20 20 F F F Max gen: 11, Max flush: 21 Begin loop 6 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.296674852Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:49.303910944Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.303968346Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.310943146Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311210617Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311246012Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.31125764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311269718Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182} {"msg":"Crucible 1cc2d07e-95ff-40b8-9c2e-af80b363379b has session id: 61dcd68a-0c0f-4bb4-a3f5-52dadac94a90","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311563145Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Upstairs opts: Upstairs UUID: 1cc2d07e-95ff-40b8-9c2e-af80b363379b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311587932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Crucible stats registered with UUID: 1cc2d07e-95ff-40b8-9c2e-af80b363379b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.31159782Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:49.311606839Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311764606Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"io task","client":"1","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311786653Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"io task","client":"2","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311796741Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"io task","client":"0","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311909745Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"task":"crutest"} {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311934321Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182} {"msg":"1cc2d07e-95ff-40b8-9c2e-af80b363379b active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311953228Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311962027Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"client":"0","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311971746Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"client":"1","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311981324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"client":"2","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.311990803Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"io task","client":"2","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.31201049Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"io task","client":"0","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.312020509Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"io task","client":"1","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"ds_connection connected from Ok(127.0.0.1:49180)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.312038376Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"io task","client":"2","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"ds_connection connected from Ok(127.0.0.1:33057)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.312109526Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"io task","client":"0","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"ds_connection connected from Ok(127.0.0.1:49937)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.312153449Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"io task","client":"1","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.312756151Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"client":"2","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.312782427Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"client":"1","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.312843988Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"client":"0","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313093332Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313137295Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"[0]R flush_numbers[0..12]: [20, 10, 7, 1, 11, 21, 21, 21, 19, 17, 17, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.31317302Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"[0]R generation[0..12]: [10, 6, 4, 1, 6, 11, 11, 11, 10, 9, 9, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313184469Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313195537Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"[1]R flush_numbers[0..12]: [20, 10, 7, 1, 11, 21, 21, 21, 19, 17, 17, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313205795Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"[1]R generation[0..12]: [10, 6, 4, 1, 6, 11, 11, 11, 10, 9, 9, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313218614Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313229862Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"[2]R flush_numbers[0..12]: [20, 10, 7, 1, 11, 21, 21, 21, 19, 17, 17, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.31324032Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"[2]R generation[0..12]: [10, 6, 4, 1, 6, 11, 11, 11, 10, 9, 9, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313250429Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313261327Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313270646Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Generation requested: 12 >= found:12","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313280195Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Next flush: 22","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313289423Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313298662Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"":"downstairs","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.31330838Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313317219Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"client":"0","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313326848Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"client":"1","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:10:49.313336416Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"client":"2","":"downstairs","session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313349104Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313358243Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"1cc2d07e-95ff-40b8-9c2e-af80b363379b is now active with session: 61dcd68a-0c0f-4bb4-a3f5-52dadac94a90","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313369421Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.31337835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313385739Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182,"session_id":"61dcd68a-0c0f-4bb4-a3f5-52dadac94a90"} {"msg":"Activated sub_volume 1cc2d07e-95ff-40b8-9c2e-af80b363379b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:49.313394818Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 316 len 16384 data: 4 4 4 4 02/30 Read block 220 len 36864 03/30 Write block 365 len 12288 data: 2 2 2 04/30 Read block 168 len 20480 05/30 Read block 87 len 40960 06/30 Read block 139 len 12288 07/30 Read block 71 len 24576 08/30 Flush 09/30 Write block 241 len 4096 data: 2 10/30 Read block 427 len 28672 11/30 Write block 422 len 24576 data: 4 3 3 3 3 3 12/30 Write block 159 len 36864 data: 3 3 3 4 4 4 4 3 2 13/30 Read block 394 len 20480 14/30 Write block 133 len 4096 data: 3 15/30 Write block 305 len 8192 data: 3 3 16/30 Read block 194 len 28672 17/30 Write block 345 len 16384 data: 3 3 3 3 18/30 Write block 588 len 24576 data: 5 4 3 3 3 2 19/30 Write block 310 len 12288 data: 2 2 2 20/30 Write block 251 len 8192 data: 2 2 21/30 Flush 22/30 Flush 23/30 Write block 338 len 40960 data: 3 3 4 4 4 3 3 4 4 4 24/30 Read block 336 len 20480 25/30 Write block 288 len 24576 data: 2 2 2 3 3 3 26/30 Write block 421 len 8192 data: 3 5 27/30 Write block 376 len 40960 data: 2 2 2 2 2 2 3 3 3 2 28/30 Read block 299 len 28672 29/30 Write block 588 len 40960 data: 6 5 4 4 4 3 2 2 2 2 30/30 Read block 306 len 40960 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Sent Done Done false 1001 Acked Read 9 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Read 6 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Write 6 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Write 2 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 10 Sent Done Done false 1029 Acked Read 10 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:49.322581425Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1182} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 20 20 20 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 4 4 4 7 7 7 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 11 12 12 21 23 23 F F F <--- 7 140-159 11 12 12 21 23 23 F F F <--- 8 160-179 10 12 12 19 23 23 F F F <--- 9 180-199 9 9 9 17 17 17 F F F 10 200-219 9 9 9 17 17 17 F F F 11 220-239 4 4 4 7 7 7 F F F 12 240-259 1 12 12 1 23 23 F F F <--- 13 260-279 10 10 10 20 20 20 F F F 14 280-299 10 10 10 20 20 20 F T T <--- 15 300-319 8 12 12 15 23 23 F F F <--- 16 320-339 10 10 10 18 18 18 F T T <--- 17 340-359 11 12 12 21 23 23 F T T <--- 18 360-379 11 12 12 21 22 22 F T T <--- 19 380-399 7 7 7 13 13 13 F T T <--- 20 400-419 11 11 11 21 21 21 F F F 21 420-439 6 12 12 10 23 23 F T T <--- 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 6 6 6 10 10 10 F F F 25 500-519 8 8 8 15 15 15 F F F 26 520-539 11 11 11 21 21 21 F F F 27 540-559 10 10 10 19 19 19 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 10 12 12 20 23 23 F T T <--- Max gen: 12, Max flush: 23 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 20 20 20 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 4 4 4 7 7 7 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 12 12 12 23 23 23 F F F 7 140-159 12 12 12 23 23 23 F F F 8 160-179 12 12 12 23 23 23 F F F 9 180-199 9 9 9 17 17 17 F F F 10 200-219 9 9 9 17 17 17 F F F 11 220-239 4 4 4 7 7 7 F F F 12 240-259 12 12 12 23 23 23 F F F 13 260-279 10 10 10 20 20 20 F F F 14 280-299 13 13 13 24 24 24 F F F 15 300-319 12 12 12 23 23 23 F F F 16 320-339 13 13 13 24 24 24 F F F 17 340-359 13 13 13 24 24 24 F F F 18 360-379 13 13 13 24 24 24 F F F 19 380-399 13 13 13 24 24 24 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 13 13 13 24 24 24 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 6 6 6 10 10 10 F F F 25 500-519 8 8 8 15 15 15 F F F 26 520-539 11 11 11 21 21 21 F F F 27 540-559 10 10 10 19 19 19 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 13 13 13 24 24 24 F F F Max gen: 13, Max flush: 24 Begin loop 7 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.577408601Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.585652046Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.585946873Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.585982568Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.585991647Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586003215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190} {"msg":"Crucible 881cb892-fef6-4629-bae8-f15f69dcf0a1 has session id: ebf58e50-3562-4226-8bc2-296773d25e20","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586282714Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Upstairs opts: Upstairs UUID: 881cb892-fef6-4629-bae8-f15f69dcf0a1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586326318Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Crucible stats registered with UUID: 881cb892-fef6-4629-bae8-f15f69dcf0a1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586337036Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:53.586346185Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586527118Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"io task","client":"0","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586566573Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586577641Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"io task","client":"2","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586783511Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"task":"crutest"} {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586804128Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190} {"msg":"881cb892-fef6-4629-bae8-f15f69dcf0a1 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586841212Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586850591Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"client":"0","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.58686021Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"client":"1","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586870078Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"client":"2","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586879587Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"io task","client":"2","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586889375Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"io task","client":"0","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586905933Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"ds_connection connected from Ok(127.0.0.1:62178)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586921571Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"io task","client":"2","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"ds_connection connected from Ok(127.0.0.1:33122)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586931939Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"io task","client":"0","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"ds_connection connected from Ok(127.0.0.1:55200)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.586972223Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.587642705Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"client":"2","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.587753439Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"client":"1","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.587772326Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"client":"0","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.587935442Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588012751Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"[0]R flush_numbers[0..12]: [20, 10, 7, 1, 11, 21, 23, 23, 23, 17, 17, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588107277Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"[0]R generation[0..12]: [10, 6, 4, 1, 6, 11, 12, 12, 12, 9, 9, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588118206Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588129104Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"[1]R flush_numbers[0..12]: [20, 10, 7, 1, 11, 21, 23, 23, 23, 17, 17, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588139433Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"[1]R generation[0..12]: [10, 6, 4, 1, 6, 11, 12, 12, 12, 9, 9, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588149591Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.58816067Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"[2]R flush_numbers[0..12]: [20, 10, 7, 1, 11, 21, 23, 23, 23, 17, 17, 7]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588171048Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"[2]R generation[0..12]: [10, 6, 4, 1, 6, 11, 12, 12, 12, 9, 9, 4]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588181167Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588192355Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Max found gen is 14","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588201664Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588211322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Next flush: 25","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588220541Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588231379Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"":"downstairs","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588241188Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588250016Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"client":"0","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588259925Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190Wait for a query_work_queue command to finish before sending IO ,"client":"1","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588273903Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"client":"2","":"downstairs","session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588283832Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.58829292Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"881cb892-fef6-4629-bae8-f15f69dcf0a1 is now active with session: ebf58e50-3562-4226-8bc2-296773d25e20","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588302639Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588311657Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588319106Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190,"session_id":"ebf58e50-3562-4226-8bc2-296773d25e20"} {"msg":"Activated sub_volume 881cb892-fef6-4629-bae8-f15f69dcf0a1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:53.588328085Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 250 len 12288 data: 2 3 3 02/30 Write block 163 len 28672 data: 5 5 5 4 3 2 2 03/30 Read block 453 len 40960 04/30 Write block 120 len 12288 data: 4 3 3 05/30 Write block 238 len 32768 data: 2 2 2 3 2 2 2 2 06/30 Write block 222 len 28672 data: 2 2 2 2 2 2 2 07/30 Write block 490 len 24576 data: 2 2 2 2 2 3 08/30 Read block 134 len 28672 09/30 Write block 320 len 16384 data: 4 4 4 3 10/30 Read block 521 len 28672 11/30 Write block 349 len 28672 data: 4 4 4 5 4 4 4 12/30 Write block 537 len 28672 data: 4 2 2 2 2 2 2 13/30 Write block 374 len 24576 data: 2 2 3 3 3 3 14/30 Write block 423 len 4096 data: 4 15/30 Read block 291 len 36864 16/30 Write block 140 len 28672 data: 5 5 4 3 2 3 3 17/30 Flush 18/30 Write block 45 len 20480 data: 3 3 3 3 3 19/30 Write block 383 len 4096 data: 4 20/30 Flush 21/30 Write block 56 len 24576 data: 2 2 2 2 2 2 22/30 Write block 259 len 40960 data: 2 2 3 3 4 4 4 4 4 4 23/30 Read block 100 len 4096 24/30 Write block 482 len 12288 data: 2 2 2 25/30 Read block 217 len 4096 26/30 Read block 452 len 20480 27/30 Write block 573 len 16384 data: 2 3 4 4 28/30 Read block 317 len 24576 29/30 Read block 263 len 32768 30/30 Read block 13 len 12288 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Write 7 Done Sent Done false 1002 Acked Read 10 Done Sent Done false 1003 Acked Write 3 Done Sent Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Read 7 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Read 1 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 1 31 Done 30 0 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:53.596814235Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1190} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 20 20 20 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 14 4 14 26 7 26 T F T <--- 3 060-079 1 1 1 1 1 1 T F T <--- 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 14 12 14 25 23 25 F F F <--- 7 140-159 14 12 14 25 23 25 F F F <--- 8 160-179 14 12 14 25 23 25 F F F <--- 9 180-199 9 9 9 17 17 17 F F F 10 200-219 9 9 9 17 17 17 F F F 11 220-239 14 4 14 25 7 25 F F F <--- 12 240-259 14 12 14 25 23 25 T F T <--- 13 260-279 10 10 10 20 20 20 T F T <--- 14 280-299 13 13 13 24 24 24 F F F 15 300-319 12 12 12 23 23 23 F F F 16 320-339 14 13 14 25 24 25 F F F <--- 17 340-359 14 13 14 25 24 25 F F F <--- 18 360-379 14 13 14 25 24 25 F F F <--- 19 380-399 14 13 14 26 24 26 F F F <--- 20 400-419 11 11 11 21 21 21 F F F 21 420-439 14 13 14 25 24 25 F F F <--- 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 14 6 14 25 10 25 T F T <--- 25 500-519 8 8 8 15 15 15 F F F 26 520-539 14 11 14 25 21 25 F F F <--- 27 540-559 14 10 14 25 19 25 F F F <--- 28 560-579 4 4 4 7 7 7 T F T <--- 29 580-599 13 13 13 24 24 24 F F F Max gen: 14, Max flush: 26 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 10 10 20 20 20 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 15 15 15 27 27 27 F F F 3 060-079 15 15 15 27 27 27 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 14 14 14 25 25 25 F F F 7 140-159 14 14 14 25 25 25 F F F 8 160-179 14 14 14 25 25 25 F F F 9 180-199 9 9 9 17 17 17 F F F 10 200-219 9 9 9 17 17 17 F F F 11 220-239 14 14 14 25 25 25 F F F 12 240-259 15 15 15 27 27 27 F F F 13 260-279 15 15 15 27 27 27 F F F 14 280-299 13 13 13 24 24 24 F F F 15 300-319 12 12 12 23 23 23 F F F 16 320-339 14 14 14 25 25 25 F F F 17 340-359 14 14 14 25 25 25 F F F 18 360-379 14 14 14 25 25 25 F F F 19 380-399 14 14 14 26 26 26 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 14 14 14 25 25 25 F F F 22 440-459 3 3 3 3 3 3 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 8 8 8 15 15 15 F F F 26 520-539 14 14 14 25 25 25 F F F 27 540-559 14 14 14 25 25 25 F F F 28 560-579 15 15 15 27 27 27 F F F 29 580-599 13 13 13 24 24 24 F F F Max gen: 15, Max flush: 27 Begin loop 8 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.852655592Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:57.859940597Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.860003418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.866990977Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.867259767Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.867287113Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.867298072Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.86730959Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198} {"msg":"Crucible df0c2593-243f-42db-b219-f0f05e4557ac has session id: 6f59b4cb-12a7-4a44-8514-abb94d17a99c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.86765032Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Upstairs opts: Upstairs UUID: df0c2593-243f-42db-b219-f0f05e4557ac, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.867670387Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Crucible stats registered with UUID: df0c2593-243f-42db-b219-f0f05e4557ac","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.867680486Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:57.867689495Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.867841032Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.867862939Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.867872918Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868000549Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"task":"crutest"} {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868017677Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198} {"msg":"df0c2593-243f-42db-b219-f0f05e4557ac active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868050782Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.86806009Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"client":"0","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868069719Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"client":"1","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868080937Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"client":"2","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868090446Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868100385Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868110193Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"ds_connection connected from Ok(127.0.0.1:57845)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868164425Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"ds_connection connected from Ok(127.0.0.1:55796)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868190551Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"ds_connection connected from Ok(127.0.0.1:42436)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.86820191Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868869762Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"client":"2","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868931753Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"client":"1","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.868943351Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"client":"0","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869219201Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869238478Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"[0]R flush_numbers[0..12]: [20, 10, 27, 27, 11, 21, 25, 25, 25, 17, 17, 25]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.86929366Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"[0]R generation[0..12]: [10, 6, 15, 15, 6, 11, 14, 14, 14, 9, 9, 14]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869304529Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869315327Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"[1]R flush_numbers[0..12]: [20, 10, 27, 27, 11, 21, 25, 25, 25, 17, 17, 25]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869327645Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"[1]R generation[0..12]: [10, 6, 15, 15, 6, 11, 14, 14, 14, 9, 9, 14]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869337684Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869348542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"[2]R flush_numbers[0..12]: [20, 10, 27, 27, 11, 21, 25, 25, 25, 17, 17, 25]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869358881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"[2]R generation[0..12]: [10, 6, 15, 15, 6, 11, 14, 14, 14, 9, 9, 14]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869368909Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869379648Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869388976Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869398515Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Next flush: 28","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869407643Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869416802Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"":"downstairs","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869426271Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869434919Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"client":"0","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869444438Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"client":"1","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869453987Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"client":"2","":"downstairs","session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869463575Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Set Downstairs and Upstairs active after reconciliation"Wait for a query_work_queue command to finish before sending IO ,"v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869477393Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"df0c2593-243f-42db-b219-f0f05e4557ac is now active with session: 6f59b4cb-12a7-4a44-8514-abb94d17a99c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869487512Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.86949658Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869503999Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198,"session_id":"6f59b4cb-12a7-4a44-8514-abb94d17a99c"} {"msg":"Activated sub_volume df0c2593-243f-42db-b219-f0f05e4557ac","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:10:57.869512938Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 273 len 20480 data: 2 2 3 3 3 02/30 Read block 474 len 16384 03/30 Read block 354 len 40960 04/30 Write block 243 len 16384 data: 3 3 3 2 05/30 Read block 498 len 36864 06/30 Write block 434 len 28672 data: 2 2 2 2 2 2 2 07/30 Write block 269 len 16384 data: 4 3 3 2 08/30 Flush 09/30 Write block 228 len 12288 data: 3 2 2 10/30 Write block 165 len 8192 data: 6 5 11/30 Read block 44 len 32768 12/30 Read block 147 len 40960 13/30 Write block 576 len 24576 data: 5 3 3 3 2 2 14/30 Flush 15/30 Write block 132 len 8192 data: 2 4 16/30 Read block 201 len 24576 17/30 Read block 206 len 24576 18/30 Write block 4 len 36864 data: 2 2 2 2 2 2 3 3 3 19/30 Flush 20/30 Write block 197 len 24576 data: 2 3 3 3 3 2 21/30 Write block 516 len 28672 data: 3 3 2 2 2 3 3 22/30 Write block 216 len 16384 data: 3 3 3 3 23/30 Read block 564 len 36864 24/30 Write block 297 len 24576 data: 2 2 2 2 2 2 25/30 Write block 261 len 36864 data: 4 4 5 5 5 5 5 5 5 26/30 Write block 5 len 12288 data: 3 3 3 27/30 Read block 186 len 40960 28/30 Read block 28 len 8192 29/30 Write block 303 len 32768 data: 3 3 4 4 2 2 2 3 30/30 Write block 181 len 20480 data: 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 4 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Write 7 Done Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 3 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Read 8 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Write 9 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Write 6 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 2 2 28 Done 6 28 28 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:10:57.878515682Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1198} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 10 16 16 20 30 30 F T T <--- 1 020-039 6 6 6 10 10 10 F F F 2 040-059 15 15 15 27 27 27 F F F 3 060-079 15 15 15 27 27 27 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 14 16 16 25 30 30 F F F <--- 7 140-159 14 14 14 25 25 25 F F F 8 160-179 14 16 16 25 29 29 F F F <--- 9 180-199 9 9 9 17 17 17 F T T <--- 10 200-219 9 9 9 17 17 17 F T T <--- 11 220-239 14 16 16 25 29 29 F F F <--- 12 240-259 15 16 16 27 28 28 T F F <--- 13 260-279 15 16 16 27 28 28 T T T <--- 14 280-299 13 13 13 24 24 24 F T T <--- 15 300-319 12 12 12 23 23 23 F T T <--- 16 320-339 14 14 14 25 25 25 F F F 17 340-359 14 14 14 25 25 25 F F F 18 360-379 14 14 14 25 25 25 F F F 19 380-399 14 14 14 26 26 26 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 14 16 16 25 28 28 T F F <--- 22 440-459 3 16 16 3 28 28 T F F <--- 23 460-479 4 4 4 5 5 5 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 8 8 8 15 15 15 F T T <--- 26 520-539 14 14 14 25 25 25 F T T <--- 27 540-559 14 14 14 25 25 25 F F F 28 560-579 15 16 16 27 29 29 F F F <--- 29 580-599 13 16 16 24 29 29 F F F <--- Max gen: 16, Max flush: 30 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 17 17 17 31 31 31 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 15 15 15 27 27 27 F F F 3 060-079 15 15 15 27 27 27 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 16 16 16 30 30 30 F F F 7 140-159 14 14 14 25 25 25 F F F 8 160-179 16 16 16 29 29 29 F F F 9 180-199 17 17 17 31 31 31 F F F 10 200-219 17 17 17 31 31 31 F F F 11 220-239 16 16 16 29 29 29 F F F 12 240-259 16 16 16 28 28 28 F F F 13 260-279 17 17 17 31 31 31 F F F 14 280-299 17 17 17 31 31 31 F F F 15 300-319 17 17 17 31 31 31 F F F 16 320-339 14 14 14 25 25 25 F F F 17 340-359 14 14 14 25 25 25 F F F 18 360-379 14 14 14 25 25 25 F F F 19 380-399 14 14 14 26 26 26 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 16 16 16 28 28 28 F F F 22 440-459 16 16 16 28 28 28 F F F 23 460-479 4 4 4 5 5 5 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 17 17 17 31 31 31 F F F 26 520-539 17 17 17 31 31 31 F F F 27 540-559 14 14 14 25 25 25 F F F 28 560-579 16 16 16 29 29 29 F F F 29 580-599 16 16 16 29 29 29 F F F Max gen: 17, Max flush: 31 Begin loop 9 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.14250157Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.150859328Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151152615Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151182241Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.15119117Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151202958Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206} {"msg":"Crucible 907ff8e7-baeb-4738-9464-f901267bed40 has session id: 2d6474b3-aec3-4fdf-908d-0769f794c38c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151574514Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Upstairs opts: Upstairs UUID: 907ff8e7-baeb-4738-9464-f901267bed40, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151603Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Crucible stats registered with UUID: 907ff8e7-baeb-4738-9464-f901267bed40","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151616378Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:02.151627706Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151777134Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151815169Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151826117Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151905135Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"task":"crutest"} {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151924523Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206} {"msg":"907ff8e7-baeb-4738-9464-f901267bed40 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151960857Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151970216Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"client":"0","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151979784Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"client":"1","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151989493Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"client":"2","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.151999042Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.15200902Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.152018959Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"ds_connection connected from Ok(127.0.0.1:42630)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.152070361Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"ds_connection connected from Ok(127.0.0.1:52001)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.15208118Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"ds_connection connected from Ok(127.0.0.1:53218)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.152091498Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.152661415Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"client":"0","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.152737084Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"client":"2","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.152748782Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"client":"1","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153016573Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153062356Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"[0]R flush_numbers[0..12]: [31, 10, 27, 27, 11, 21, 30, 25, 29, 31, 31, 29]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153092812Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"[0]R generation[0..12]: [17, 6, 15, 15, 6, 11, 16, 14, 16, 17, 17, 16]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.15310456Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153115718Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"[1]R flush_numbers[0..12]: [31, 10, 27, 27, 11, 21, 30, 25, 29, 31, 31, 29]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153126247Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"[1]R generation[0..12]: [17, 6, 15, 15, 6, 11, 16, 14, 16, 17, 17, 16]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153136505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153147324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"[2]R flush_numbers[0..12]: [31, 10, 27, 27, 11, 21, 30, 25, 29, 31, 31, 29]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153157672Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"[2]R generation[0..12]: [17, 6, 15, 15, 6, 11, 16, 14, 16, 17, 17, 16]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153167881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153178829Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Max found gen is 18","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153189838Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Generation requested: 18 >= found:18","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153199476Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Next flush: 32","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153208745Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153218074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"":"downstairs","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153227622Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153236271Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"client":"0","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153245799Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"client":"1","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153255428Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"client":"2","":"downstairs","session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153265177Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:11:02.153274275Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"907ff8e7-baeb-4738-9464-f901267bed40 is now active with session: 2d6474b3-aec3-4fdf-908d-0769f794c38c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153286823Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153295802Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.153303371Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206,"session_id":"2d6474b3-aec3-4fdf-908d-0769f794c38c"} {"msg":"Activated sub_volume 907ff8e7-baeb-4738-9464-f901267bed40","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:02.15331246Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 548 len 40960 data: 4 4 4 4 3 3 3 3 2 2 02/30 Read block 20 len 32768 03/30 Write block 454 len 40960 data: 2 2 3 3 3 3 3 3 3 3 04/30 Read block 511 len 8192 05/30 Flush 06/30 Write block 54 len 16384 data: 2 2 3 3 07/30 Read block 171 len 36864 08/30 Read block 130 len 16384 09/30 Read block 332 len 40960 10/30 Flush 11/30 Write block 238 len 40960 data: 3 3 3 4 3 4 4 4 3 2 12/30 Write block 311 len 32768 data: 3 3 3 3 3 5 5 5 13/30 Write block 273 len 8192 data: 3 3 14/30 Flush 15/30 Write block 434 len 12288 data: 3 3 3 16/30 Write block 454 len 24576 data: 3 3 4 4 4 4 17/30 Read block 544 len 32768 18/30 Flush 19/30 Write block 358 len 28672 data: 4 4 5 5 2 2 2 20/30 Read block 143 len 20480 21/30 Read block 377 len 36864 22/30 Write block 279 len 12288 data: 3 3 3 23/30 Write block 581 len 20480 data: 3 2 2 2 3 24/30 Write block 12 len 12288 data: 4 3 3 25/30 Write block 167 len 40960 data: 4 3 3 2 2 2 2 2 2 2 26/30 Read block 293 len 36864 27/30 Write block 505 len 12288 data: 2 2 2 28/30 Write block 458 len 8192 data: 5 5 29/30 Write block 445 len 20480 data: 4 4 4 4 3 30/30 Write block 442 len 4096 data: 4 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 8 Done Sent Done false 1002 Acked Write 10 Done Sent Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 4 Done Sent Done false 1006 Acked Read 9 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Write 2 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 2 Sent Sent Sent false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 29 3 35 Done 27 1 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) None Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:02.162024696Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1206} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 17 17 17 31 31 31 T F T <--- 1 020-039 6 6 6 10 10 10 F F F 2 040-059 18 15 18 33 27 33 F F F <--- 3 060-079 15 15 15 27 27 27 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 16 16 16 30 30 30 F F F 7 140-159 14 14 14 25 25 25 F F F 8 160-179 16 16 16 29 29 29 T F T <--- 9 180-199 17 17 17 31 31 31 F F F 10 200-219 17 17 17 31 31 31 F F F 11 220-239 18 16 18 34 29 34 F F F <--- 12 240-259 18 16 18 34 28 34 F F F <--- 13 260-279 18 17 18 34 31 34 T F T <--- 14 280-299 17 17 17 31 31 31 T F T <--- 15 300-319 18 17 18 34 31 34 F F F <--- 16 320-339 14 14 14 25 25 25 F F F 17 340-359 14 14 14 25 25 25 T F T <--- 18 360-379 14 14 14 25 25 25 T F T <--- 19 380-399 14 14 14 26 26 26 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 18 16 18 35 28 35 F F F <--- 22 440-459 18 16 18 35 28 35 T F T <--- 23 460-479 18 4 18 32 5 32 F F F <--- 24 480-499 15 15 15 27 27 27 F F F 25 500-519 17 17 17 31 31 31 T F T <--- 26 520-539 17 17 17 31 31 31 F F F 27 540-559 18 14 18 32 25 32 F T F <--- 28 560-579 16 16 16 29 29 29 F F F 29 580-599 16 16 16 29 29 29 T F T <--- Max gen: 18, Max flush: 35 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 19 19 19 36 36 36 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 18 18 18 33 33 33 F F F 3 060-079 15 15 15 27 27 27 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 16 16 16 30 30 30 F F F 7 140-159 14 14 14 25 25 25 F F F 8 160-179 19 19 19 36 36 36 F F F 9 180-199 17 17 17 31 31 31 F F F 10 200-219 17 17 17 31 31 31 F F F 11 220-239 18 18 18 34 34 34 F F F 12 240-259 18 18 18 34 34 34 F F F 13 260-279 19 19 19 36 36 36 F F F 14 280-299 19 19 19 36 36 36 F F F 15 300-319 18 18 18 34 34 34 F F F 16 320-339 14 14 14 25 25 25 F F F 17 340-359 19 19 19 36 36 36 F F F 18 360-379 19 19 19 36 36 36 F F F 19 380-399 14 14 14 26 26 26 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 18 18 18 35 35 35 F F F 22 440-459 19 19 19 36 36 36 F F F 23 460-479 18 18 18 32 32 32 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 19 19 19 36 36 36 F F F 26 520-539 17 17 17 31 31 31 F F F 27 540-559 18 18 18 32 32 32 F F F 28 560-579 16 16 16 29 29 29 F F F 29 580-599 19 19 19 36 36 36 F F F Max gen: 19, Max flush: 36 Begin loop 10 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.417600513Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.425751152Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426046368Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426074294Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426084603Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426096031Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214} {"msg":"Crucible 4cf0dae5-2372-460a-9035-adb5bfa96e21 has session id: 4550cd91-6bda-43bf-b536-45822e63dac5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426413345Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Upstairs opts: Upstairs UUID: 4cf0dae5-2372-460a-9035-adb5bfa96e21, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426437971Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Crucible stats registered with UUID: 4cf0dae5-2372-460a-9035-adb5bfa96e21","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426448Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:06.426459238Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426632173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426669287Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"io task","client":"0","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426680296Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"io task","client":"2","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.42678682Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"task":"crutest"} {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426811837Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214} {"msg":"4cf0dae5-2372-460a-9035-adb5bfa96e21 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426836073Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426845292Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"client":"0","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.42685494Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"client":"1","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426866319Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"client":"2","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426875867Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"io task","client":"2","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426894984Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"ds_connection connected from Ok(127.0.0.1:49176)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426936838Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"io task","client":"2","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"ds_connection connected from Ok(127.0.0.1:39959)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426952146Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.426965324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"io task","client":"0","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"ds_connection connected from Ok(127.0.0.1:53008)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.427081367Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"io task","client":"0","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.427595392Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"client":"0","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.427622138Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"client":"2","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.427647444Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"client":"1","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.427933233Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.427944391Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"[0]R flush_numbers[0..12]: [36, 10, 33, 27, 11, 21, 30, 25, 36, 31, 31, 34]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.427982136Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"[0]R generation[0..12]: [19, 6, 18, 15, 6, 11, 16, 14, 19, 17, 17, 18]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.427993084Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428004302Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"[1]R flush_numbers[0..12]: [36, 10, 33, 27, 11, 21, 30, 25, 36, 31, 31, 34]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.42801744Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"[1]R generation[0..12]: [19, 6, 18, 15, 6, 11, 16, 14, 19, 17, 17, 18]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428027949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428038767Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"[2]R flush_numbers[0..12]: [36, 10, 33, 27, 11, 21, 30, 25, 36, 31, 31, 34]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428049006Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"[2]R generation[0..12]: [19, 6, 18, 15, 6, 11, 16, 14, 19, 17, 17, 18]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428059124Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428069973Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Max found gen is 20","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428079211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Generation requested: 20 >= found:20","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.42808869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Next flush: 37","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428097969Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428107287Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"":"downstairs","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428116906Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428125505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"client":"0","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428135083Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"client":"1","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428144542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"client":"2","":"downstairs","session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.42815419Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Wait for a query_work_queue command to finish before sending IO Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428168628Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"4cf0dae5-2372-460a-9035-adb5bfa96e21 is now active with session: 4550cd91-6bda-43bf-b536-45822e63dac5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428178687Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428187685Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428195074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214,"session_id":"4550cd91-6bda-43bf-b536-45822e63dac5"} {"msg":"Activated sub_volume 4cf0dae5-2372-460a-9035-adb5bfa96e21","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:06.428204113Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 466 len 24576 data: 2 2 2 2 2 2 03/30 Read block 305 len 8192 04/30 Read block 439 len 12288 05/30 Read block 175 len 8192 06/30 Write block 306 len 4096 data: 5 07/30 Write block 340 len 28672 data: 5 5 5 4 4 5 5 08/30 Write block 440 len 24576 data: 3 4 5 4 4 5 09/30 Write block 327 len 32768 data: 2 2 2 3 3 3 3 3 10/30 Write block 296 len 36864 data: 2 3 3 3 3 3 3 4 4 11/30 Flush 12/30 Write block 283 len 12288 data: 3 3 2 13/30 Write block 345 len 4096 data: 6 14/30 Read block 14 len 32768 15/30 Flush 16/30 Flush 17/30 Write block 528 len 20480 data: 5 6 5 4 4 18/30 Read block 181 len 12288 19/30 Write block 350 len 32768 data: 5 5 6 5 5 5 4 4 20/30 Read block 404 len 4096 21/30 Read block 453 len 36864 22/30 Write block 513 len 40960 data: 2 4 3 4 4 3 3 3 4 4 23/30 Write block 537 len 28672 data: 5 3 3 3 3 3 3 24/30 Write block 1 len 40960 data: 2 2 2 3 4 4 4 3 3 4 25/30 Write block 139 len 36864 data: 4 6 6 5 4 3 4 4 3 26/30 Write block 537 len 12288 data: 6 4 4 27/30 Read block 40 len 40960 28/30 Read block 84 len 4096 29/30 Write block 45 len 16384 data: 4 4 4 4 30/30 Read block 91 len 24576 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:19 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1011 Acked Write 3 Done Done Done false 1012 Acked Write 1 Done Done Done false 1013 Acked Read 8 Done Done Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Write 4 Done Sent Done false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 16 1 17 Done 19 3 18 40 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1010)) Some(JobId(1015)) Downstairs last five completed: 1010 1009 1008 1007 1006 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:06.436683134Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1214} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 19 19 19 36 36 36 T F T <--- 1 020-039 6 6 6 10 10 10 F F F 2 040-059 18 18 18 33 33 33 T F T <--- 3 060-079 15 15 15 27 27 27 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 16 16 16 30 30 30 T F T <--- 7 140-159 14 14 14 25 25 25 T F T <--- 8 160-179 19 19 19 36 36 36 F F F 9 180-199 17 17 17 31 31 31 F F F 10 200-219 17 17 17 31 31 31 F F F 11 220-239 18 18 18 34 34 34 F F F 12 240-259 18 18 18 34 34 34 F F F 13 260-279 19 19 19 36 36 36 F F F 14 280-299 20 20 20 39 38 39 F T F <--- 15 300-319 20 20 20 38 38 38 F F F 16 320-339 20 20 20 38 38 38 F F F 17 340-359 20 20 20 39 38 39 T T T <--- 18 360-379 19 19 19 36 36 36 F F F 19 380-399 14 14 14 26 26 26 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 18 18 18 35 35 35 F F F 22 440-459 20 20 20 38 38 38 F F F 23 460-479 20 20 20 38 38 38 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 19 19 19 36 36 36 T F T <--- 26 520-539 17 17 17 31 31 31 T F T <--- 27 540-559 18 18 18 32 32 32 T F T <--- 28 560-579 16 16 16 29 29 29 F F F 29 580-599 19 19 19 36 36 36 F F F Max gen: 20, Max flush: 39 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 21 21 21 40 40 40 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 21 21 21 40 40 40 F F F 3 060-079 15 15 15 27 27 27 F F F 4 080-099 6 6 6 11 11 11 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 21 21 21 40 40 40 F F F 7 140-159 21 21 21 40 40 40 F F F 8 160-179 19 19 19 36 36 36 F F F 9 180-199 17 17 17 31 31 31 F F F 10 200-219 17 17 17 31 31 31 F F F 11 220-239 18 18 18 34 34 34 F F F 12 240-259 18 18 18 34 34 34 F F F 13 260-279 19 19 19 36 36 36 F F F 14 280-299 20 20 20 39 39 39 F F F 15 300-319 20 20 20 38 38 38 F F F 16 320-339 20 20 20 38 38 38 F F F 17 340-359 21 21 21 40 40 40 F F F 18 360-379 19 19 19 36 36 36 F F F 19 380-399 14 14 14 26 26 26 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 18 18 18 35 35 35 F F F 22 440-459 20 20 20 38 38 38 F F F 23 460-479 20 20 20 38 38 38 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 21 21 21 40 40 40 F F F 26 520-539 21 21 21 40 40 40 F F F 27 540-559 21 21 21 40 40 40 F F F 28 560-579 16 16 16 29 29 29 F F F 29 580-599 19 19 19 36 36 36 F F F Max gen: 21, Max flush: 40 Begin loop 11 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.681192108Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:10.688395365Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.688454456Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.695534792Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.695810981Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.695842247Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.695851405Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.695863644Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222} {"msg":"Crucible 5e29419b-3049-4b64-8ba8-6ffdb0613cf6 has session id: 413ea950-a05c-432b-8166-8dcac440f9fe","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696198455Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Upstairs opts: Upstairs UUID: 5e29419b-3049-4b64-8ba8-6ffdb0613cf6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.69623085Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Crucible stats registered with UUID: 5e29419b-3049-4b64-8ba8-6ffdb0613cf6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696242308Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:10.696251667Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696385507Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"io task","client":"2","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696407974Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"io task","client":"0","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696418242Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696537885Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"task":"crutest"} {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696563541Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222} {"msg":"5e29419b-3049-4b64-8ba8-6ffdb0613cf6 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696580489Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696589287Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"client":"0","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696598876Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"client":"1","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696608505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"client":"2","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696618013Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"io task","client":"2","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696628002Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"io task","client":"0","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.69663808Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"ds_connection connected from Ok(127.0.0.1:34127)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696698162Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"io task","client":"2","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"ds_connection connected from Ok(127.0.0.1:55258)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696724158Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"ds_connection connected from Ok(127.0.0.1:56261)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.696735686Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"io task","client":"0","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697269788Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"client":"1","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697355515Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"client":"2","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697368104Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"client":"0","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697534539Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697642404Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"[0]R flush_numbers[0..12]: [40, 10, 40, 27, 11, 21, 40, 40, 36, 31, 31, 34]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697687637Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"[0]R generation[0..12]: [21, 6, 21, 15, 6, 11, 21, 21, 19, 17, 17, 18]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697698645Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697709634Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"[1]R flush_numbers[0..12]: [40, 10, 40, 27, 11, 21, 40, 40, 36, 31, 31, 34]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697720172Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"[1]R generation[0..12]: [21, 6, 21, 15, 6, 11, 21, 21, 19, 17, 17, 18]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697730611Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697741459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"[2]R flush_numbers[0..12]: [40, 10, 40, 27, 11, 21, 40, 40, 36, 31, 31, 34]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697751828Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"[2]R generation[0..12]: [21, 6, 21, 15, 6, 11, 21, 21, 19, 17, 17, 18]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697762056Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697772964Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697782453Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697793631Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Next flush: 41","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.69780302Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697812579Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"":"downstairs","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697822567Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697831396Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"client":"0","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697841574Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"client":"1","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697855802Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"client":"2","":"downstairs","session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697865831Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.69787503Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"5e29419b-3049-4b64-8ba8-6ffdb0613cf6 is now active with session: 413ea950-a05c-432b-8166-8dcac440f9fe","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697884898Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697893967Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697901526Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222,"session_id":"413ea950-a05c-432b-8166-8dcac440f9fe"} {"msg":"Activated sub_volume 5e29419b-3049-4b64-8ba8-6ffdb0613cf6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:10.697910724Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 318 len 12288 data: 6 5 5 02/30 Read block 437 len 28672 03/30 Read block 544 len 24576 04/30 Write block 186 len 24576 data: 2 2 2 2 2 2 05/30 Read block 383 len 40960 06/30 Write block 350 len 32768 data: 6 6 7 6 6 6 5 5 07/30 Read block 400 len 20480 08/30 Write block 370 len 32768 data: 2 2 2 2 3 3 4 4 09/30 Write block 81 len 28672 data: 3 3 3 3 4 3 3 10/30 Write block 320 len 36864 data: 6 5 5 4 3 2 2 3 3 11/30 Write block 584 len 8192 data: 3 4 12/30 Write block 508 len 28672 data: 2 2 2 2 2 3 5 13/30 Read block 334 len 36864 14/30 Read block 188 len 20480 15/30 Write block 139 len 28672 data: 5 7 7 6 5 4 5 16/30 Read block 339 len 12288 17/30 Write block 238 len 12288 data: 4 4 4 18/30 Read block 255 len 24576 19/30 Write block 224 len 36864 data: 3 3 3 3 4 3 3 2 2 20/30 Read block 222 len 20480 21/30 Read block 383 len 36864 22/30 Read block 184 len 28672 23/30 Flush 24/30 Write block 346 len 36864 data: 6 5 4 5 7 7 8 7 7 25/30 Write block 441 len 24576 data: 5 6 5 5 6 5 26/30 Read block 507 len 40960 27/30 Write block 325 len 12288 data: 3 3 4 28/30 Read block 571 len 8192 29/30 Read block 285 len 32768 30/30 Write block 521 len 16384 data: 5 5 4 4 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Write 6 Done Done Done false 1004 Acked Read 10 Done Done Done false 1005 Acked Write 8 Done Done Done false 1006 Acked Read 5 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Write 7 Sent Done Done false 1009 Acked Write 9 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Read 9 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 1 1 26 Done 6 29 29 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:10.70731241Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1222} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 21 21 21 40 40 40 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 21 21 21 40 40 40 F F F 3 060-079 15 15 15 27 27 27 F F F 4 080-099 6 22 22 11 41 41 F F F <--- 5 100-119 11 11 11 21 21 21 F F F 6 120-139 21 22 22 40 41 41 F F F <--- 7 140-159 21 22 22 40 41 41 F F F <--- 8 160-179 19 19 19 36 36 36 F F F 9 180-199 17 22 22 31 41 41 T F F <--- 10 200-219 17 17 17 31 31 31 F F F 11 220-239 18 22 22 34 41 41 F F F <--- 12 240-259 18 22 22 34 41 41 F F F <--- 13 260-279 19 19 19 36 36 36 F F F 14 280-299 20 20 20 39 39 39 F F F 15 300-319 20 22 22 38 41 41 T F F <--- 16 320-339 20 22 22 38 41 41 T T T <--- 17 340-359 21 22 22 40 41 41 T T T <--- 18 360-379 19 22 22 36 41 41 F F F <--- 19 380-399 14 14 14 26 26 26 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 18 18 18 35 35 35 F F F 22 440-459 20 20 20 38 38 38 F T T <--- 23 460-479 20 20 20 38 38 38 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 21 22 22 40 41 41 F F F <--- 26 520-539 21 21 21 40 40 40 F T T <--- 27 540-559 21 21 21 40 40 40 F F F 28 560-579 16 16 16 29 29 29 F F F 29 580-599 19 22 22 36 41 41 F F F <--- Max gen: 22, Max flush: 41 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 21 21 21 40 40 40 F F F 1 020-039 6 6 6 10 10 10 F F F 2 040-059 21 21 21 40 40 40 F F F 3 060-079 15 15 15 27 27 27 F F F 4 080-099 22 22 22 41 41 41 F F F 5 100-119 11 11 11 21 21 21 F F F 6 120-139 22 22 22 41 41 41 F F F 7 140-159 22 22 22 41 41 41 F F F 8 160-179 19 19 19 36 36 36 F F F 9 180-199 22 22 22 41 41 41 F F F 10 200-219 17 17 17 31 31 31 F F F 11 220-239 22 22 22 41 41 41 F F F 12 240-259 22 22 22 41 41 41 F F F 13 260-279 19 19 19 36 36 36 F F F 14 280-299 20 20 20 39 39 39 F F F 15 300-319 22 22 22 41 41 41 F F F 16 320-339 23 23 23 42 42 42 F F F 17 340-359 23 23 23 42 42 42 F F F 18 360-379 22 22 22 41 41 41 F F F 19 380-399 14 14 14 26 26 26 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 18 18 18 35 35 35 F F F 22 440-459 23 23 23 42 42 42 F F F 23 460-479 20 20 20 38 38 38 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 22 22 22 41 41 41 F F F 26 520-539 23 23 23 42 42 42 F F F 27 540-559 21 21 21 40 40 40 F F F 28 560-579 16 16 16 29 29 29 F F F 29 580-599 22 22 22 41 41 41 F F F Max gen: 23, Max flush: 42 Begin loop 12 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.964911931Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.973228605Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.973523852Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.973550078Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.973560746Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.973572215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230} {"msg":"Crucible f712741c-298e-443c-928e-ccdeba763f8e has session id: 0f8320cf-d3b5-4236-9f21-fbc4f39b241b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.973898617Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Upstairs opts: Upstairs UUID: f712741c-298e-443c-928e-ccdeba763f8e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.973922364Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Crucible stats registered with UUID: f712741c-298e-443c-928e-ccdeba763f8e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.973932232Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:14.973941251Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974110966Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974147781Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"io task","client":"0","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974158909Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"io task","client":"1","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974250226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"task":"crutest"} {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974274942Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230} {"msg":"f712741c-298e-443c-928e-ccdeba763f8e active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974300728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974310227Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"client":"0","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974319955Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"client":"1","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974331364Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"client":"2","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974340822Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974350551Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"io task","client":"0","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.97436044Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"io task","client":"1","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"ds_connection connected from Ok(127.0.0.1:54765)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974460205Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"ds_connection connected from Ok(127.0.0.1:43748)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974485431Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"io task","client":"1","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"ds_connection connected from Ok(127.0.0.1:47185)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.974544263Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"io task","client":"0","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975167532Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"client":"2","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975204976Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"client":"1","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975221864Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"client":"0","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975466688Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975493964Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"[0]R flush_numbers[0..12]: [40, 10, 40, 27, 41, 21, 41, 41, 36, 41, 31, 41]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975561694Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"[0]R generation[0..12]: [21, 6, 21, 15, 22, 11, 22, 22, 19, 22, 17, 22]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975573112Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975584061Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"[1]R flush_numbers[0..12]: [40, 10, 40, 27, 41, 21, 41, 41, 36, 41, 31, 41]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975595749Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"[1]R generation[0..12]: [21, 6, 21, 15, 22, 11, 22, 22, 19, 22, 17, 22]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975606167Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975617096Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"[2]R flush_numbers[0..12]: [40, 10, 40, 27, 41, 21, 41, 41, 36, 41, 31, 41]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975627434Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"[2]R generation[0..12]: [21, 6, 21, 15, 22, 11, 22, 22, 19, 22, 17, 22]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975637603Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975648421Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.9756577Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975670278Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Next flush: 43","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975679557Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975688775Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"":"downstairs","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975698404Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975707053Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"client":"0","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975716601Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"client":"1","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.97572619Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"client":"2","":"downstairs","session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975740858Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975749846Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"f712741c-298e-443c-928e-ccdeba763f8e is now active with session: 0f8320cf-d3b5-4236-9f21-fbc4f39b241b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975759725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975768694Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975776153Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230,"session_id":"0f8320cf-d3b5-4236-9f21-fbc4f39b241b"} {"msg":"Activated sub_volume f712741c-298e-443c-928e-ccdeba763f8e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:14.975785251Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 102 len 4096 data: 3 02/30 Write block 129 len 8192 data: 2 2 03/30 Read block 105 len 36864 04/30 Read block 75 len 28672 05/30 Read block 342 len 40960 06/30 Read block 172 len 28672 07/30 Read block 126 len 40960 08/30 Write block 15 len 28672 data: 3 3 3 3 3 2 2 09/30 Write block 442 len 36864 data: 7 6 6 7 6 5 5 4 2 10/30 Read block 546 len 4096 11/30 Write block 466 len 36864 data: 3 3 3 3 3 3 2 3 3 12/30 Read block 396 len 4096 13/30 Write block 141 len 36864 data: 8 7 6 5 6 5 4 3 3 14/30 Read block 425 len 20480 15/30 Flush 16/30 Write block 223 len 16384 data: 3 4 4 4 17/30 Write block 350 len 36864 data: 8 8 9 8 8 7 6 6 5 18/30 Read block 32 len 20480 19/30 Write block 178 len 36864 data: 2 2 2 3 3 3 3 3 3 20/30 Read block 596 len 12288 21/30 Write block 589 len 24576 data: 6 5 5 5 4 3 22/30 Write block 99 len 12288 data: 4 4 4 23/30 Read block 101 len 20480 24/30 Read block 430 len 8192 25/30 Read block 521 len 36864 26/30 Write block 114 len 40960 data: 4 4 4 3 3 3 5 4 4 3 27/30 Write block 384 len 32768 data: 4 3 2 2 2 3 3 3 28/30 Read block 152 len 40960 29/30 Write block 38 len 12288 data: 2 2 2 30/30 Write block 271 len 16384 data: 4 3 4 4 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 9 Done Sent Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Read 7 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Read 1 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 4 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Read 3 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 28 2 32 Done 28 2 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) None Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:14.984877022Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1230} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 21 24 43 40 43 F F F <--- 1 020-039 24 6 24 43 10 43 T F T <--- 2 040-059 21 21 21 40 40 40 T F T <--- 3 060-079 15 15 15 27 27 27 F F F 4 080-099 22 22 22 41 41 41 T F T <--- 5 100-119 24 11 24 43 21 43 T T T <--- 6 120-139 24 22 24 43 41 43 T T T <--- 7 140-159 24 22 24 43 41 43 F F F <--- 8 160-179 19 19 19 36 36 36 T F T <--- 9 180-199 22 22 22 41 41 41 T F T <--- 10 200-219 17 17 17 31 31 31 F F F 11 220-239 22 22 22 41 41 41 T F T <--- 12 240-259 22 22 22 41 41 41 F F F 13 260-279 19 19 19 36 36 36 T F T <--- 14 280-299 20 20 20 39 39 39 F F F 15 300-319 22 22 22 41 41 41 F F F 16 320-339 23 23 23 42 42 42 F F F 17 340-359 23 23 23 42 42 42 T F T <--- 18 360-379 22 22 22 41 41 41 F F F 19 380-399 14 14 14 26 26 26 T F T <--- 20 400-419 11 11 11 21 21 21 F F F 21 420-439 18 18 18 35 35 35 F F F 22 440-459 24 23 24 43 42 43 F F F <--- 23 460-479 24 20 24 43 38 43 F F F <--- 24 480-499 15 15 15 27 27 27 F F F 25 500-519 22 22 22 41 41 41 F F F 26 520-539 23 23 23 42 42 42 F F F 27 540-559 21 21 21 40 40 40 F F F 28 560-579 16 16 16 29 29 29 F F F 29 580-599 22 22 22 41 41 41 T F T <--- Max gen: 24, Max flush: 43 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 43 43 43 F F F 1 020-039 25 25 25 44 44 44 F F F 2 040-059 25 25 25 44 44 44 F F F 3 060-079 15 15 15 27 27 27 F F F 4 080-099 25 25 25 44 44 44 F F F 5 100-119 25 25 25 44 44 44 F F F 6 120-139 25 25 25 44 44 44 F F F 7 140-159 24 24 24 43 43 43 F F F 8 160-179 25 25 25 44 44 44 F F F 9 180-199 25 25 25 44 44 44 F F F 10 200-219 17 17 17 31 31 31 F F F 11 220-239 25 25 25 44 44 44 F F F 12 240-259 22 22 22 41 41 41 F F F 13 260-279 25 25 25 44 44 44 F F F 14 280-299 20 20 20 39 39 39 F F F 15 300-319 22 22 22 41 41 41 F F F 16 320-339 23 23 23 42 42 42 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 22 22 22 41 41 41 F F F 19 380-399 25 25 25 44 44 44 F F F 20 400-419 11 11 11 21 21 21 F F F 21 420-439 18 18 18 35 35 35 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 24 24 24 43 43 43 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 22 22 22 41 41 41 F F F 26 520-539 23 23 23 42 42 42 F F F 27 540-559 21 21 21 40 40 40 F F F 28 560-579 16 16 16 29 29 29 F F F 29 580-599 25 25 25 44 44 44 F F F Max gen: 25, Max flush: 44 Begin loop 13 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.242818683Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:19.249994174Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.250057495Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.257093907Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.257355619Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.257381585Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.257390193Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.257401542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238} {"msg":"Crucible dd809c90-f35d-4a5d-9bcd-99d70691573a has session id: e1311d4c-d4bc-42d9-9465-f8be81f449c3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.257705547Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Upstairs opts: Upstairs UUID: dd809c90-f35d-4a5d-9bcd-99d70691573a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.257733863Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Crucible stats registered with UUID: dd809c90-f35d-4a5d-9bcd-99d70691573a","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.257746771Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:19.25775583Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.2578901Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"io task","client":"2","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.257914577Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.257924735Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"io task","client":"1","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258024741Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"task":"crutest"} {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258048967Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238} {"msg":"dd809c90-f35d-4a5d-9bcd-99d70691573a active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258073764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258083002Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"client":"0","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258092561Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"client":"1","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258102139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"client":"2","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258111588Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"io task","client":"2","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258121257Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258134545Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"io task","client":"1","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:44622)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258264516Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"io task","client":"1","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:38744)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258319658Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"io task","client":"2","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:45463)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258344654Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258866858Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"client":"2","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258891714Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"client":"1","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.258902642Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"client":"0","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259208378Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259254301Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"[0]R flush_numbers[0..12]: [43, 44, 44, 27, 44, 44, 44, 43, 44, 44, 31, 44]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259309623Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"[0]R generation[0..12]: [24, 25, 25, 15, 25, 25, 25, 24, 25, 25, 17, 25]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259321171Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.25933254Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"[1]R flush_numbers[0..12]: [43, 44, 44, 27, 44, 44, 44, 43, 44, 44, 31, 44]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259343168Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"[1]R generation[0..12]: [24, 25, 25, 15, 25, 25, 25, 24, 25, 25, 17, 25]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259353437Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259364395Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"[2]R flush_numbers[0..12]: [43, 44, 44, 27, 44, 44, 44, 43, 44, 44, 31, 44]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259374643Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"[2]R generation[0..12]: [24, 25, 25, 15, 25, 25, 25, 24, 25, 25, 17, 25]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259385012Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.25939795Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259407519Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259417147Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Next flush: 45","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259426246Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259435505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"":"downstairs","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259445073Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259453572Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"client":"0","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.25946316Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"client":Wait for a query_work_queue command to finish before sending IO "1","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259477718Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"client":"2","":"downstairs","session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259487737Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259496856Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"dd809c90-f35d-4a5d-9bcd-99d70691573a is now active with session: e1311d4c-d4bc-42d9-9465-f8be81f449c3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259506694Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259515693Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.259523062Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238,"session_id":"e1311d4c-d4bc-42d9-9465-f8be81f449c3"} {"msg":"Activated sub_volume dd809c90-f35d-4a5d-9bcd-99d70691573a","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:19.25953212Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 144 len 8192 data: 6 7 02/30 Write block 72 len 28672 data: 2 2 2 2 2 2 2 03/30 Write block 169 len 20480 data: 4 3 3 3 3 04/30 Flush 05/30 Write block 556 len 36864 data: 3 3 2 2 3 3 3 3 3 06/30 Write block 289 len 40960 data: 3 3 4 4 4 3 2 3 4 4 07/30 Read block 327 len 28672 08/30 Read block 83 len 28672 09/30 Flush 10/30 Write block 404 len 32768 data: 2 2 3 3 3 2 3 3 11/30 Read block 260 len 32768 12/30 Write block 565 len 32768 data: 3 3 3 2 2 2 2 2 13/30 Flush 14/30 Write block 292 len 12288 data: 5 5 4 15/30 Read block 427 len 8192 16/30 Write block 246 len 12288 data: 4 3 2 17/30 Flush 18/30 Write block 71 len 36864 data: 2 3 3 3 3 3 3 3 2 19/30 Read block 118 len 12288 20/30 Read block 594 len 16384 21/30 Write block 536 len 32768 data: 6 7 5 5 4 4 4 4 22/30 Write block 386 len 36864 data: 3 3 3 4 4 4 3 3 2 23/30 Write block 234 len 40960 data: 2 2 2 2 5 5 5 5 4 5 24/30 Write block 548 len 32768 data: 5 5 5 5 4 4 4 4 25/30 Read block 292 len 20480 26/30 Read block 168 len 16384 27/30 Read block 508 len 32768 28/30 Read block 101 len 36864 29/30 Read block 280 len 40960 30/30 Read block 4 len 16384 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Write 5 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Write 10 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Read 8 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 3 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Write 3 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 9 Sent Done Done false 1018 Acked Read 3 Sent Done Done false 1019 Acked Read 4 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 1 29 Done 2 30 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:19.26877396Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1238} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 43 43 43 F F F 1 020-039 25 25 25 44 44 44 F F F 2 040-059 25 25 25 44 44 44 F F F 3 060-079 15 26 26 27 45 45 T T T <--- 4 080-099 25 25 25 44 44 44 F F F 5 100-119 25 25 25 44 44 44 F F F 6 120-139 25 25 25 44 44 44 F F F 7 140-159 24 26 26 43 45 45 T F F <--- 8 160-179 25 26 26 44 45 45 F F F <--- 9 180-199 25 25 25 44 44 44 F F F 10 200-219 17 17 17 31 31 31 F F F 11 220-239 25 25 25 44 44 44 F T T <--- 12 240-259 22 26 26 41 48 48 F T T <--- 13 260-279 25 25 25 44 44 44 F F F 14 280-299 20 26 26 39 48 48 F F F <--- 15 300-319 22 22 22 41 41 41 F F F 16 320-339 23 23 23 42 42 42 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 22 22 22 41 41 41 F F F 19 380-399 25 25 25 44 44 44 F T T <--- 20 400-419 11 26 26 21 47 47 F F F <--- 21 420-439 18 18 18 35 35 35 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 24 24 24 43 43 43 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 22 22 22 41 41 41 F F F 26 520-539 23 23 23 42 42 42 F T T <--- 27 540-559 21 26 26 40 46 46 F T T <--- 28 560-579 16 26 26 29 47 47 F F F <--- 29 580-599 25 25 25 44 44 44 F F F Max gen: 26, Max flush: 48 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 13 Verifying data now No change after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 43 43 43 F F F 1 020-039 25 25 25 44 44 44 F F F 2 040-059 25 25 25 44 44 44 F F F 3 060-079 27 27 27 49 49 49 F F F 4 080-099 25 25 25 44 44 44 F F F 5 100-119 25 25 25 44 44 44 F F F 6 120-139 25 25 25 44 44 44 F F F 7 140-159 26 26 26 45 45 45 F F F 8 160-179 26 26 26 45 45 45 F F F 9 180-199 25 25 25 44 44 44 F F F 10 200-219 17 17 17 31 31 31 F F F 11 220-239 27 27 27 49 49 49 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 25 25 25 44 44 44 F F F 14 280-299 26 26 26 48 48 48 F F F 15 300-319 22 22 22 41 41 41 F F F 16 320-339 23 23 23 42 42 42 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 22 22 22 41 41 41 F F F 19 380-399 27 27 27 49 49 49 F F F 20 400-419 26 26 26 47 47 47 F F F 21 420-439 18 18 18 35 35 35 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 24 24 24 43 43 43 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 22 22 22 41 41 41 F F F 26 520-539 27 27 27 49 49 49 F F F 27 540-559 27 27 27 49 49 49 F F F 28 560-579 26 26 26 47 47 47 F F F 29 580-599 25 25 25 44 44 44 F F F Max gen: 27, Max flush: 49 Begin loop 14 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.520467364Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.528799496Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529087394Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529121399Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529130597Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529142236Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246} {"msg":"Crucible 468b356b-a5be-4197-b5b2-fd87119e2418 has session id: a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529493944Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Upstairs opts: Upstairs UUID: 468b356b-a5be-4197-b5b2-fd87119e2418, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529519771Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Crucible stats registered with UUID: 468b356b-a5be-4197-b5b2-fd87119e2418","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529529929Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:23.529539038Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529715012Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529737169Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"io task","client":"0","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529747127Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"io task","client":"1","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529852262Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"task":"crutest"} {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529870429Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246} {"msg":"468b356b-a5be-4197-b5b2-fd87119e2418 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529901725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529912853Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"client":"0","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529922372Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"client":"1","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.52993182Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"client":"2","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529941519Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.529951517Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"io task","client":"1","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.530031326Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"io task","client":"0","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"ds_connection connected from Ok(127.0.0.1:63718)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.530058682Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"io task","client":"1","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"ds_connection connected from Ok(127.0.0.1:58087)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.53006937Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"ds_connection connected from Ok(127.0.0.1:52929)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.530079569Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"io task","client":"0","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.530732333Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"client":"0","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.53075739Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"client":"1","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.530809682Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"client":"2","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531062505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531083872Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"[0]R flush_numbers[0..12]: [43, 44, 44, 49, 44, 44, 44, 45, 45, 44, 31, 49]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.53109427Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"[0]R generation[0..12]: [24, 25, 25, 27, 25, 25, 25, 26, 26, 25, 17, 27]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531107469Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531118537Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"[1]R flush_numbers[0..12]: [43, 44, 44, 49, 44, 44, 44, 45, 45, 44, 31, 49]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531128835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"[1]R generation[0..12]: [24, 25, 25, 27, 25, 25, 25, 26, 26, 25, 17, 27]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531138994Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531149632Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"[2]R flush_numbers[0..12]: [43, 44, 44, 49, 44, 44, 44, 45, 45, 44, 31, 49]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531159991Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"[2]R generation[0..12]: [24, 25, 25, 27, 25, 25, 25, 26, 26, 25, 17, 27]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531170209Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531180908Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Max found gen is 28","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531190226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Generation requested: 28 >= found:28","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531199715Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Next flush: 50","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531208874Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531218052Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"":"downstairs","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531227691Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.53123638Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"client":"0","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531246028Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"client":"1","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531256577Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"client":"2","":"downstairs","session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531266285Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531275354Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pidWait for a query_work_queue command to finish before sending IO ":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"468b356b-a5be-4197-b5b2-fd87119e2418 is now active with session: a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531289812Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531298771Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531306519Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246,"session_id":"a89ee8f9-ed6e-46b5-8ae1-5177c895b3b1"} {"msg":"Activated sub_volume 468b356b-a5be-4197-b5b2-fd87119e2418","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:23.531315538Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 143 len 40960 data: 7 7 8 6 5 4 4 3 3 3 02/30 Write block 24 len 8192 data: 2 3 03/30 Read block 113 len 40960 04/30 Write block 554 len 16384 data: 5 5 4 4 05/30 Flush 06/30 Write block 82 len 4096 data: 4 07/30 Read block 220 len 24576 08/30 Write block 84 len 16384 data: 4 5 4 4 09/30 Write block 36 len 24576 data: 2 2 3 3 3 2 10/30 Write block 32 len 28672 data: 2 2 2 2 3 3 4 11/30 Write block 466 len 28672 data: 4 4 4 4 4 4 3 12/30 Read block 351 len 4096 13/30 Read block 38 len 12288 14/30 Read block 209 len 12288 15/30 Write block 286 len 12288 data: 2 2 3 16/30 Read block 142 len 8192 17/30 Write block 221 len 40960 data: 2 3 4 5 5 5 4 5 4 4 18/30 Read block 338 len 28672 19/30 Flush 20/30 Write block 419 len 8192 data: 4 3 21/30 Read block 419 len 8192 22/30 Read block 67 len 16384 23/30 Write block 204 len 8192 data: 3 3 24/30 Write block 572 len 36864 data: 3 3 4 5 6 4 4 4 3 25/30 Read block 306 len 20480 26/30 Read block 479 len 4096 27/30 Read block 485 len 40960 28/30 Read block 442 len 28672 29/30 Write block 372 len 16384 data: 3 3 4 4 30/30 Read block 514 len 40960 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Sent false 1001 Acked Write 2 Done Done Sent false 1002 Acked Read 10 Done Done Sent false 1003 Acked Write 4 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Read 6 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 6 Done Done Sent false 1009 Acked Write 7 Done Done Sent false 1010 Acked Write 7 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Read 3 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Write 9 Done Done Sent false 1024 Acked Read 5 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Read 10 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 10 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:23.540313003Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1246} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 43 43 43 F F F 1 020-039 28 28 25 51 51 44 F F F <--- 2 040-059 28 28 25 51 51 44 F F F <--- 3 060-079 27 27 27 49 49 49 F F F 4 080-099 28 28 25 51 51 44 F F F <--- 5 100-119 25 25 25 44 44 44 F F F 6 120-139 25 25 25 44 44 44 F F F 7 140-159 28 28 26 50 50 45 F F F <--- 8 160-179 26 26 26 45 45 45 F F F 9 180-199 25 25 25 44 44 44 F F F 10 200-219 17 17 17 31 31 31 T T F <--- 11 220-239 28 28 27 51 51 49 F F F <--- 12 240-259 27 27 27 49 49 49 F F F 13 260-279 25 25 25 44 44 44 F F F 14 280-299 28 28 26 51 51 48 F F F <--- 15 300-319 22 22 22 41 41 41 F F F 16 320-339 23 23 23 42 42 42 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 22 22 22 41 41 41 T T F <--- 19 380-399 27 27 27 49 49 49 F F F 20 400-419 26 26 26 47 47 47 T T F <--- 21 420-439 18 18 18 35 35 35 T T F <--- 22 440-459 24 24 24 43 43 43 F F F 23 460-479 28 28 24 51 51 43 F F F <--- 24 480-499 15 15 15 27 27 27 F F F 25 500-519 22 22 22 41 41 41 F F F 26 520-539 27 27 27 49 49 49 F F F 27 540-559 28 28 27 50 50 49 F F F <--- 28 560-579 26 26 26 47 47 47 T T F <--- 29 580-599 25 25 25 44 44 44 T T F <--- Max gen: 28, Max flush: 51 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 43 43 43 F F F 1 020-039 28 28 28 51 51 51 F F F 2 040-059 28 28 28 51 51 51 F F F 3 060-079 27 27 27 49 49 49 F F F 4 080-099 28 28 28 51 51 51 F F F 5 100-119 25 25 25 44 44 44 F F F 6 120-139 25 25 25 44 44 44 F F F 7 140-159 28 28 28 50 50 50 F F F 8 160-179 26 26 26 45 45 45 F F F 9 180-199 25 25 25 44 44 44 F F F 10 200-219 29 29 29 52 52 52 F F F 11 220-239 28 28 28 51 51 51 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 25 25 25 44 44 44 F F F 14 280-299 28 28 28 51 51 51 F F F 15 300-319 22 22 22 41 41 41 F F F 16 320-339 23 23 23 42 42 42 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 29 29 29 52 52 52 F F F 19 380-399 27 27 27 49 49 49 F F F 20 400-419 29 29 29 52 52 52 F F F 21 420-439 29 29 29 52 52 52 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 28 28 28 51 51 51 F F F 24 480-499 15 15 15 27 27 27 F F F 25 500-519 22 22 22 41 41 41 F F F 26 520-539 27 27 27 49 49 49 F F F 27 540-559 28 28 28 50 50 50 F F F 28 560-579 29 29 29 52 52 52 F F F 29 580-599 29 29 29 52 52 52 F F F Max gen: 29, Max flush: 52 Begin loop 15 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.791093151Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:27.798301407Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.798356979Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.805208857Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.805492306Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.805529591Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.805538929Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.805550577Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254} {"msg":"Crucible ceadf239-0557-499f-bcdf-86375e0a54fd has session id: c628939f-7a35-4a71-bb1f-aa54fb63f1ab","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.805854493Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Upstairs opts: Upstairs UUID: ceadf239-0557-499f-bcdf-86375e0a54fd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.80587771Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Crucible stats registered with UUID: ceadf239-0557-499f-bcdf-86375e0a54fd","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.805887838Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:27.805896567Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806052424Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"io task","client":"1","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806088129Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806098897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"io task","client":"2","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806206522Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"task":"crutest"} {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806223439Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254} {"msg":"ceadf239-0557-499f-bcdf-86375e0a54fd active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806263163Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806272632Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"client":"0","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806282221Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"client":"1","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806291779Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"client":"2","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806301298Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"io task","client":"2","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806311256Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806321085Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"io task","client":"1","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:55173)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806363369Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"io task","client":"2","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:55140)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806374297Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:53854)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.806384446Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"io task","client":"1","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807061667Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"client":"2","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807114759Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"client":"1","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807227012Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"client":"0","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807343145Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807359093Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"[0]R flush_numbers[0..12]: [43, 51, 51, 49, 51, 44, 44, 50, 45, 44, 52, 51]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807501332Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"[0]R generation[0..12]: [24, 28, 28, 27, 28, 25, 25, 28, 26, 25, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807512311Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807523259Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"[1]R flush_numbers[0..12]: [43, 51, 51, 49, 51, 44, 44, 50, 45, 44, 52, 51]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807533728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"[1]R generation[0..12]: [24, 28, 28, 27, 28, 25, 25, 28, 26, 25, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807544096Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807554965Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"[2]R flush_numbers[0..12]: [43, 51, 51, 49, 51, 44, 44, 50, 45, 44, 52, 51]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807567233Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"[2]R generation[0..12]: [24, 28, 28, 27, 28, 25, 25, 28, 26, 25, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807577421Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.80758815Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807597798Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807607407Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Next flush: 53","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807616636Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807625974Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"":"downstairs","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807635583Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807644241Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"client":"0","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.80765383Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"client":"1","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807663349Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"client":"2","":"downstairs","session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807673307Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:11:27.807682386Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"ceadf239-0557-499f-bcdf-86375e0a54fd is now active with session: c628939f-7a35-4a71-bb1f-aa54fb63f1ab","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807696364Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807705533Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.807713291Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254,"session_id":"c628939f-7a35-4a71-bb1f-aa54fb63f1ab"} {"msg":"Activated sub_volume ceadf239-0557-499f-bcdf-86375e0a54fd","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:27.80772339Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 279 len 40960 data: 4 4 4 3 4 4 3 3 3 4 02/30 Read block 455 len 32768 03/30 Write block 4 len 40960 data: 4 5 5 5 4 4 5 4 5 4 04/30 Write block 395 len 16384 data: 2 2 2 2 05/30 Read block 372 len 24576 06/30 Write block 374 len 4096 data: 5 07/30 Read block 484 len 32768 08/30 Write block 87 len 24576 data: 5 3 3 3 3 2 09/30 Write block 39 len 8192 data: 4 4 10/30 Read block 39 len 40960 11/30 Read block 559 len 20480 12/30 Read block 184 len 16384 13/30 Write block 584 len 8192 data: 4 5 14/30 Write block 101 len 28672 data: 5 4 3 3 4 5 6 15/30 Read block 143 len 36864 16/30 Write block 533 len 12288 data: 5 5 6 17/30 Flush 18/30 Write block 489 len 36864 data: 2 3 3 3 3 3 4 3 3 19/30 Flush 20/30 Write block 192 len 40960 data: 2 2 2 2 2 3 4 4 4 4 21/30 Write block 304 len 12288 data: 5 5 6 22/30 Read block 41 len 16384 23/30 Flush 24/30 Write block 323 len 36864 data: 5 4 4 4 5 4 3 4 4 25/30 Write block 410 len 36864 data: 4 4 3 3 3 3 3 4 4 26/30 Flush 27/30 Write block 218 len 8192 data: 4 4 28/30 Read block 589 len 40960 29/30 Write block 28 len 20480 data: 4 4 3 2 3 30/30 Write block 316 len 8192 data: 6 6 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Sent Done Done false 1001 Acked Read 8 Sent Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Write 4 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Read 4 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Write 7 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Write 3 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 9 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 2 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:27.817350303Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1254} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 30 30 43 53 53 F F F <--- 1 020-039 28 30 30 51 53 53 F T T <--- 2 040-059 28 30 30 51 53 53 F F F <--- 3 060-079 27 27 27 49 49 49 F F F 4 080-099 28 30 30 51 53 53 F F F <--- 5 100-119 25 30 30 44 53 53 F F F <--- 6 120-139 25 25 25 44 44 44 F F F 7 140-159 28 28 28 50 50 50 F F F 8 160-179 26 26 26 45 45 45 F F F 9 180-199 25 30 30 44 55 55 F F F <--- 10 200-219 29 30 30 52 55 55 F T T <--- 11 220-239 28 28 28 51 51 51 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 25 30 30 44 53 53 F F F <--- 14 280-299 28 30 30 51 53 53 F F F <--- 15 300-319 22 30 30 41 55 55 F T T <--- 16 320-339 23 30 30 42 56 56 F F F <--- 17 340-359 25 25 25 44 44 44 F F F 18 360-379 29 30 30 52 53 53 F F F <--- 19 380-399 27 30 30 49 53 53 F F F <--- 20 400-419 29 30 30 52 56 56 F F F <--- 21 420-439 29 29 29 52 52 52 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 28 28 28 51 51 51 F F F 24 480-499 15 30 30 27 54 54 F F F <--- 25 500-519 22 22 22 41 41 41 F F F 26 520-539 27 30 30 49 53 53 F F F <--- 27 540-559 28 28 28 50 50 50 F F F 28 560-579 29 29 29 52 52 52 F F F 29 580-599 29 30 30 52 53 53 F F F <--- Max gen: 30, Max flush: 56 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 30 30 30 53 53 53 F F F 1 020-039 31 31 31 57 57 57 F F F 2 040-059 30 30 30 53 53 53 F F F 3 060-079 27 27 27 49 49 49 F F F 4 080-099 30 30 30 53 53 53 F F F 5 100-119 30 30 30 53 53 53 F F F 6 120-139 25 25 25 44 44 44 F F F 7 140-159 28 28 28 50 50 50 F F F 8 160-179 26 26 26 45 45 45 F F F 9 180-199 30 30 30 55 55 55 F F F 10 200-219 31 31 31 57 57 57 F F F 11 220-239 28 28 28 51 51 51 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 30 30 30 53 53 53 F F F 14 280-299 30 30 30 53 53 53 F F F 15 300-319 31 31 31 57 57 57 F F F 16 320-339 30 30 30 56 56 56 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 30 30 30 53 53 53 F F F 19 380-399 30 30 30 53 53 53 F F F 20 400-419 30 30 30 56 56 56 F F F 21 420-439 29 29 29 52 52 52 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 28 28 28 51 51 51 F F F 24 480-499 30 30 30 54 54 54 F F F 25 500-519 22 22 22 41 41 41 F F F 26 520-539 30 30 30 53 53 53 F F F 27 540-559 28 28 28 50 50 50 F F F 28 560-579 29 29 29 52 52 52 F F F 29 580-599 30 30 30 53 53 53 F F F Max gen: 31, Max flush: 57 Begin loop 16 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.079991095Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:32.087193882Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.087255973Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.094149986Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.094411308Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.094437584Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.094446003Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.094457381Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262} {"msg":"Crucible 66eee435-9b26-4b60-be54-ef069b3dbc5d has session id: 3c6fd3a5-c952-47f5-9370-dd56b3e244fd","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.094841115Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Upstairs opts: Upstairs UUID: 66eee435-9b26-4b60-be54-ef069b3dbc5d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.094864201Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Crucible stats registered with UUID: 66eee435-9b26-4b60-be54-ef069b3dbc5d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.09487409Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:32.094883499Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095050514Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"io task","client":"2","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095085689Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"io task","client":"0","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095099107Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095202912Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"task":"crutest"} {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095220389Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262} {"msg":"66eee435-9b26-4b60-be54-ef069b3dbc5d active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095258524Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095268072Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"client":"0","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095277791Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"client":"1","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.09528759Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"client":"2","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095297418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"io task","client":"2","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095307337Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"io task","client":"0","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095317165Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"ds_connection connected from Ok(127.0.0.1:45037)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095376447Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"ds_connection connected from Ok(127.0.0.1:46123)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095402443Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"io task","client":"2","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"ds_connection connected from Ok(127.0.0.1:52095)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095444507Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"io task","client":"0","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095916308Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"client":"2","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.09596647Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"client":"1","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.095987547Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"client":"0","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096200836Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096227592Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"[0]R flush_numbers[0..12]: [53, 57, 53, 49, 53, 53, 44, 50, 45, 55, 57, 51]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.09630763Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"[0]R generation[0..12]: [30, 31, 30, 27, 30, 30, 25, 28, 26, 30, 31, 28]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096318969Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096329957Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"[1]R flush_numbers[0..12]: [53, 57, 53, 49, 53, 53, 44, 50, 45, 55, 57, 51]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096340346Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"[1]R generation[0..12]: [30, 31, 30, 27, 30, 30, 25, 28, 26, 30, 31, 28]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096350574Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096361283Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"[2]R flush_numbers[0..12]: [53, 57, 53, 49, 53, 53, 44, 50, 45, 55, 57, 51]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096371581Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"[2]R generation[0..12]: [30, 31, 30, 27, 30, 30, 25, 28, 26, 30, 31, 28]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.09638173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096392678Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096402007Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096411645Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Next flush: 58","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096420914Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096432422Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"":"downstairs","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096442011Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.09645059Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"client":"0","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096460268Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"client":"1","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":Wait for a query_work_queue command to finish before sending IO "2026-04-15T18:11:32.096470147Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"client":"2","":"downstairs","session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096484085Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096493163Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"66eee435-9b26-4b60-be54-ef069b3dbc5d is now active with session: 3c6fd3a5-c952-47f5-9370-dd56b3e244fd","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096502982Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096512001Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096519469Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262,"session_id":"3c6fd3a5-c952-47f5-9370-dd56b3e244fd"} {"msg":"Activated sub_volume 66eee435-9b26-4b60-be54-ef069b3dbc5d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:32.096528428Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 24 len 20480 data: 3 4 4 4 5 03/30 Flush 04/30 Write block 545 len 32768 data: 3 4 4 6 6 6 6 5 05/30 Flush 06/30 Write block 180 len 32768 data: 3 4 4 4 4 4 4 3 07/30 Write block 469 len 28672 data: 5 5 5 4 4 4 2 08/30 Write block 526 len 32768 data: 4 4 6 7 6 5 5 6 09/30 Write block 283 len 16384 data: 5 5 4 4 10/30 Write block 267 len 36864 data: 6 6 6 4 5 4 5 5 4 11/30 Write block 570 len 8192 data: 3 3 12/30 Write block 157 len 36864 data: 3 3 4 4 4 5 6 6 7 13/30 Write block 206 len 32768 data: 3 2 2 2 2 2 2 2 14/30 Read block 244 len 32768 15/30 Read block 287 len 8192 16/30 Read block 357 len 16384 17/30 Read block 481 len 28672 18/30 Read block 2 len 32768 19/30 Read block 352 len 32768 20/30 Write block 327 len 24576 data: 6 5 4 5 5 4 21/30 Read block 21 len 16384 22/30 Read block 83 len 8192 23/30 Flush 24/30 Flush 25/30 Write block 331 len 36864 data: 6 5 4 4 3 4 3 4 4 26/30 Read block 425 len 20480 27/30 Read block 442 len 32768 28/30 Write block 368 len 28672 data: 2 2 3 3 4 4 6 29/30 Read block 413 len 28672 30/30 Read block 52 len 32768 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 5 Done Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 8 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 9 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Write 8 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 8 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 0 28 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:32.105856545Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1262} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 30 30 30 53 53 53 F F F 1 020-039 31 32 32 57 59 59 T F F <--- 2 040-059 30 30 30 53 53 53 F F F 3 060-079 27 27 27 49 49 49 F F F 4 080-099 30 30 30 53 53 53 F F F 5 100-119 30 30 30 53 53 53 F F F 6 120-139 25 25 25 44 44 44 F F F 7 140-159 28 32 32 50 61 61 F F F <--- 8 160-179 26 32 32 45 61 61 F F F <--- 9 180-199 30 32 32 55 61 61 F F F <--- 10 200-219 31 32 32 57 61 61 F F F <--- 11 220-239 28 28 28 51 51 51 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 30 32 32 53 61 61 F F F <--- 14 280-299 30 32 32 53 61 61 F F F <--- 15 300-319 31 31 31 57 57 57 F F F 16 320-339 30 32 32 56 61 61 F T T <--- 17 340-359 25 25 25 44 44 44 F F F 18 360-379 30 30 30 53 53 53 F T T <--- 19 380-399 30 30 30 53 53 53 F F F 20 400-419 30 30 30 56 56 56 F F F 21 420-439 29 29 29 52 52 52 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 28 32 32 51 61 61 F F F <--- 24 480-499 30 30 30 54 54 54 F F F 25 500-519 22 22 22 41 41 41 F F F 26 520-539 30 32 32 53 61 61 F F F <--- 27 540-559 28 32 32 50 60 60 F F F <--- 28 560-579 29 32 32 52 61 61 F F F <--- 29 580-599 30 30 30 53 53 53 F F F Max gen: 32, Max flush: 61 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 30 30 30 53 53 53 F F F 1 020-039 32 32 32 59 59 59 F F F 2 040-059 30 30 30 53 53 53 F F F 3 060-079 27 27 27 49 49 49 F F F 4 080-099 30 30 30 53 53 53 F F F 5 100-119 30 30 30 53 53 53 F F F 6 120-139 25 25 25 44 44 44 F F F 7 140-159 32 32 32 61 61 61 F F F 8 160-179 32 32 32 61 61 61 F F F 9 180-199 32 32 32 61 61 61 F F F 10 200-219 32 32 32 61 61 61 F F F 11 220-239 28 28 28 51 51 51 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 32 32 32 61 61 61 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 31 31 31 57 57 57 F F F 16 320-339 33 33 33 62 62 62 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 33 33 33 62 62 62 F F F 19 380-399 30 30 30 53 53 53 F F F 20 400-419 30 30 30 56 56 56 F F F 21 420-439 29 29 29 52 52 52 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 32 32 32 61 61 61 F F F 24 480-499 30 30 30 54 54 54 F F F 25 500-519 22 22 22 41 41 41 F F F 26 520-539 32 32 32 61 61 61 F F F 27 540-559 32 32 32 60 60 60 F F F 28 560-579 32 32 32 61 61 61 F F F 29 580-599 30 30 30 53 53 53 F F F Max gen: 33, Max flush: 62 Begin loop 17 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.356964134Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:36.364187268Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.364250029Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.371111056Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.371376537Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.371401963Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.371413692Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.3714252Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270} {"msg":"Crucible 7e75de81-a448-4d2c-874b-210270f1bbb3 has session id: f4947f51-a395-44a8-bfcd-e51151d8692d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.371734385Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Upstairs opts: Upstairs UUID: 7e75de81-a448-4d2c-874b-210270f1bbb3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.371756142Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Crucible stats registered with UUID: 7e75de81-a448-4d2c-874b-210270f1bbb3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.3717662Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:36.371775339Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.371924767Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.371947494Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"io task","client":"2","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.371957432Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"io task","client":"0","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372093592Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"task":"crutest"} {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.37210652Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270} {"msg":"7e75de81-a448-4d2c-874b-210270f1bbb3 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372124578Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372133427Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"client":"0","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372143015Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"client":"1","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372152544Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"client":"2","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372163752Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"io task","client":"2","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372173731Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"io task","client":"0","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372183659Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"ds_connection connected from Ok(127.0.0.1:54090)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372232092Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"io task","client":"0","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"ds_connection connected from Ok(127.0.0.1:40848)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372242991Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"io task","client":"2","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"ds_connection connected from Ok(127.0.0.1:62941)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372345056Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372942588Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"client":"1","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372966515Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"client":"0","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.372977273Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"client":"2","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373261152Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.37327088Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"[0]R flush_numbers[0..12]: [53, 59, 53, 49, 53, 53, 44, 61, 61, 61, 61, 51]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373360887Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"[0]R generation[0..12]: [30, 32, 30, 27, 30, 30, 25, 32, 32, 32, 32, 28]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373371876Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373382704Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"[1]R flush_numbers[0..12]: [53, 59, 53, 49, 53, 53, 44, 61, 61, 61, 61, 51]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373394602Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"[1]R generation[0..12]: [30, 32, 30, 27, 30, 30, 25, 32, 32, 32, 32, 28]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373404821Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373415509Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"[2]R flush_numbers[0..12]: [53, 59, 53, 49, 53, 53, 44, 61, 61, 61, 61, 51]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373425918Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"[2]R generation[0..12]: [30, 32, 30, 27, 30, 30, 25, 32, 32, 32, 32, 28]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373436236Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373446965Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Max found gen is 34","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373456263Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373465722Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Next flush: 63","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.37347487Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373484079Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"":"downstairs","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373493848Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373502436Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"client":"0","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373511945Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"client":"1","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373521474Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","Wait for a query_work_queue command to finish before sending IO pid":1270,"client":"2","":"downstairs","session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373535032Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.37354539Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"7e75de81-a448-4d2c-874b-210270f1bbb3 is now active with session: f4947f51-a395-44a8-bfcd-e51151d8692d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373555289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373564277Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373572026Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270,"session_id":"f4947f51-a395-44a8-bfcd-e51151d8692d"} {"msg":"Activated sub_volume 7e75de81-a448-4d2c-874b-210270f1bbb3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:36.373581175Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 76 len 12288 data: 4 4 4 02/30 Read block 539 len 12288 03/30 Write block 262 len 36864 data: 5 6 6 6 6 7 7 7 5 04/30 Write block 79 len 12288 data: 3 3 4 05/30 Read block 579 len 24576 06/30 Write block 143 len 28672 data: 8 8 9 7 6 5 5 07/30 Read block 550 len 36864 08/30 Read block 14 len 32768 09/30 Flush 10/30 Write block 82 len 32768 data: 5 4 5 6 5 6 4 4 11/30 Read block 510 len 16384 12/30 Write block 7 len 20480 data: 6 5 5 6 5 13/30 Write block 391 len 12288 data: 5 4 4 14/30 Read block 138 len 8192 15/30 Read block 130 len 12288 16/30 Write block 402 len 20480 data: 2 2 3 3 4 17/30 Read block 401 len 20480 18/30 Write block 237 len 20480 data: 3 6 6 6 6 19/30 Read block 155 len 40960 20/30 Write block 134 len 8192 data: 3 3 21/30 Read block 90 len 28672 22/30 Write block 163 len 8192 data: 7 7 23/30 Flush 24/30 Write block 494 len 28672 data: 4 5 4 4 3 2 2 25/30 Read block 508 len 32768 26/30 Read block 133 len 4096 27/30 Read block 263 len 24576 28/30 Write block 406 len 28672 data: 5 4 4 3 5 5 4 29/30 Write block 245 len 28672 data: 5 5 4 3 2 3 4 30/30 Write block 506 len 16384 data: 3 3 3 3 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Read 3 Sent Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Write 7 Sent Done Done false 1006 Acked Read 9 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Write 2 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Write 7 Sent Sent Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 3 3 36 Done 0 27 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:36.382864948Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1270} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 30 34 34 53 64 64 F F F <--- 1 020-039 32 32 32 59 59 59 F F F 2 040-059 30 30 30 53 53 53 F F F 3 060-079 27 34 34 49 63 63 F F F <--- 4 080-099 30 34 34 53 64 64 F F F <--- 5 100-119 30 30 30 53 53 53 F F F 6 120-139 25 34 34 44 64 64 F F F <--- 7 140-159 32 34 34 61 63 63 F F F <--- 8 160-179 32 34 34 61 64 64 F F F <--- 9 180-199 32 32 32 61 61 61 F F F 10 200-219 32 32 32 61 61 61 F F F 11 220-239 28 34 34 51 64 64 F F F <--- 12 240-259 27 34 34 49 64 64 F T T <--- 13 260-279 32 34 34 61 63 63 F F F <--- 14 280-299 32 32 32 61 61 61 F F F 15 300-319 31 31 31 57 57 57 F F F 16 320-339 33 33 33 62 62 62 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 33 33 33 62 62 62 F F F 19 380-399 30 34 34 53 64 64 F F F <--- 20 400-419 30 34 34 56 64 64 F T T <--- 21 420-439 29 29 29 52 52 52 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 32 32 32 61 61 61 F F F 24 480-499 30 30 30 54 54 54 F T T <--- 25 500-519 22 22 22 41 41 41 F T T <--- 26 520-539 32 32 32 61 61 61 F F F 27 540-559 32 32 32 60 60 60 F F F 28 560-579 32 32 32 61 61 61 F F F 29 580-599 30 30 30 53 53 53 F F F Max gen: 34, Max flush: 64 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 64 64 64 F F F 1 020-039 32 32 32 59 59 59 F F F 2 040-059 30 30 30 53 53 53 F F F 3 060-079 34 34 34 63 63 63 F F F 4 080-099 34 34 34 64 64 64 F F F 5 100-119 30 30 30 53 53 53 F F F 6 120-139 34 34 34 64 64 64 F F F 7 140-159 34 34 34 63 63 63 F F F 8 160-179 34 34 34 64 64 64 F F F 9 180-199 32 32 32 61 61 61 F F F 10 200-219 32 32 32 61 61 61 F F F 11 220-239 34 34 34 64 64 64 F F F 12 240-259 35 35 35 65 65 65 F F F 13 260-279 34 34 34 63 63 63 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 31 31 31 57 57 57 F F F 16 320-339 33 33 33 62 62 62 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 33 33 33 62 62 62 F F F 19 380-399 34 34 34 64 64 64 F F F 20 400-419 35 35 35 65 65 65 F F F 21 420-439 29 29 29 52 52 52 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 32 32 32 61 61 61 F F F 24 480-499 35 35 35 65 65 65 F F F 25 500-519 35 35 35 65 65 65 F F F 26 520-539 32 32 32 61 61 61 F F F 27 540-559 32 32 32 60 60 60 F F F 28 560-579 32 32 32 61 61 61 F F F 29 580-599 30 30 30 53 53 53 F F F Max gen: 35, Max flush: 65 Begin loop 18 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.63771535Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.64592091Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.646201519Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.646231565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.646240024Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.646251542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278} {"msg":"Crucible d8a69207-7f9a-42e3-a208-977b916568c6 has session id: 6cd1771b-1c02-47e5-8b9c-60e6d99a084f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.646599291Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Upstairs opts: Upstairs UUID: d8a69207-7f9a-42e3-a208-977b916568c6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.64667746Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Crucible stats registered with UUID: d8a69207-7f9a-42e3-a208-977b916568c6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.646703966Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:40.646714374Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.646834957Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.64688192Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"io task","client":"2","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.646905126Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"io task","client":"1","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647037507Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"task":"crutest"} {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647050285Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278} {"msg":"d8a69207-7f9a-42e3-a208-977b916568c6 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647073702Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.64708301Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"client":"0","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647092669Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"client":"1","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647102538Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"client":"2","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647112236Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"io task","client":"2","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647121885Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647131613Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"io task","client":"1","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"ds_connection connected from Ok(127.0.0.1:39937)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647266994Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"io task","client":"2","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"ds_connection connected from Ok(127.0.0.1:42705)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647343092Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"ds_connection connected from Ok(127.0.0.1:36440)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647354031Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"io task","client":"1","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647885343Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"client":"0","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.64790585Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"client":"1","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.647947554Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"client":"2","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648200477Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648231283Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"[0]R flush_numbers[0..12]: [64, 59, 53, 63, 64, 53, 64, 63, 64, 61, 61, 64]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648262298Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"[0]R generation[0..12]: [34, 32, 30, 34, 34, 30, 34, 34, 34, 32, 32, 34]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648273176Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648284105Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"[1]R flush_numbers[0..12]: [64, 59, 53, 63, 64, 53, 64, 63, 64, 61, 61, 64]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648294543Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"[1]R generation[0..12]: [34, 32, 30, 34, 34, 30, 34, 34, 34, 32, 32, 34]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648304862Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.64831576Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"[2]R flush_numbers[0..12]: [64, 59, 53, 63, 64, 53, 64, 63, 64, 61, 61, 64]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648326289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"[2]R generation[0..12]: [34, 32, 30, 34, 34, 30, 34, 34, 34, 32, 32, 34]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648336547Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648347456Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648356894Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648368253Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Next flush: 66","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648377391Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.64838687Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"":"downstairs","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648396478Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648405217Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"client":"0","":"downstairs","Wait for a query_work_queue command to finish before sending IO session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648419505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"client":"1","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648429224Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"client":"2","":"downstairs","session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648438862Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648447801Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"d8a69207-7f9a-42e3-a208-977b916568c6 is now active with session: 6cd1771b-1c02-47e5-8b9c-60e6d99a084f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648457769Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648466858Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648474647Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278,"session_id":"6cd1771b-1c02-47e5-8b9c-60e6d99a084f"} {"msg":"Activated sub_volume d8a69207-7f9a-42e3-a208-977b916568c6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:40.648483786Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 588 len 4096 data: 7 02/30 Write block 226 len 20480 data: 6 5 6 5 5 03/30 Read block 517 len 24576 04/30 Write block 150 len 24576 data: 4 4 4 3 2 2 05/30 Flush 06/30 Write block 316 len 20480 data: 7 7 7 6 7 07/30 Read block 493 len 40960 08/30 Flush 09/30 Write block 497 len 12288 data: 5 4 3 10/30 Read block 496 len 20480 11/30 Write block 582 len 24576 data: 3 3 5 6 4 5 12/30 Write block 117 len 36864 data: 4 4 4 6 5 5 4 4 3 13/30 Read block 181 len 40960 14/30 Read block 76 len 4096 15/30 Read block 376 len 36864 16/30 Read block 441 len 36864 17/30 Flush 18/30 Write block 585 len 4096 data: 7 19/30 Write block 191 len 36864 data: 3 3 3 3 3 3 4 5 5 20/30 Read block 121 len 16384 21/30 Read block 96 len 20480 22/30 Read block 74 len 20480 23/30 Read block 214 len 32768 24/30 Flush 25/30 Write block 428 len 28672 data: 3 3 2 2 2 2 4 26/30 Read block 551 len 4096 27/30 Read block 244 len 32768 28/30 Read block 2 len 24576 29/30 Write block 584 len 20480 data: 6 8 5 6 8 30/30 Write block 463 len 32768 data: 4 2 2 5 5 5 6 6 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Sent Done false 1001 Acked Write 5 Done Sent Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Write 6 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 5 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 2 34 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:40.657524394Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1278} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 64 64 64 F F F 1 020-039 32 32 32 59 59 59 F F F 2 040-059 30 30 30 53 53 53 F F F 3 060-079 34 34 34 63 63 63 F F F 4 080-099 34 34 34 64 64 64 F F F 5 100-119 36 30 36 68 53 68 F F F <--- 6 120-139 36 34 36 68 64 68 F F F <--- 7 140-159 36 34 36 66 63 66 F F F <--- 8 160-179 34 34 34 64 64 64 F F F 9 180-199 36 32 36 69 61 69 F F F <--- 10 200-219 32 32 32 61 61 61 F F F 11 220-239 36 34 36 66 64 66 F F F <--- 12 240-259 35 35 35 65 65 65 F F F 13 260-279 34 34 34 63 63 63 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 36 31 36 67 57 67 F F F <--- 16 320-339 36 33 36 67 62 67 F F F <--- 17 340-359 25 25 25 44 44 44 F F F 18 360-379 33 33 33 62 62 62 F F F 19 380-399 34 34 34 64 64 64 F F F 20 400-419 35 35 35 65 65 65 F F F 21 420-439 29 29 29 52 52 52 T F T <--- 22 440-459 24 24 24 43 43 43 F F F 23 460-479 32 32 32 61 61 61 T F T <--- 24 480-499 36 35 36 68 65 68 F F F <--- 25 500-519 35 35 35 65 65 65 F F F 26 520-539 32 32 32 61 61 61 F F F 27 540-559 32 32 32 60 60 60 F F F 28 560-579 32 32 32 61 61 61 F F F 29 580-599 36 30 36 69 53 69 T F T <--- Max gen: 36, Max flush: 69 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 64 64 64 F F F 1 020-039 32 32 32 59 59 59 F F F 2 040-059 30 30 30 53 53 53 F F F 3 060-079 34 34 34 63 63 63 F F F 4 080-099 34 34 34 64 64 64 F F F 5 100-119 36 36 36 68 68 68 F F F 6 120-139 36 36 36 68 68 68 F F F 7 140-159 36 36 36 66 66 66 F F F 8 160-179 34 34 34 64 64 64 F F F 9 180-199 36 36 36 69 69 69 F F F 10 200-219 32 32 32 61 61 61 F F F 11 220-239 36 36 36 66 66 66 F F F 12 240-259 35 35 35 65 65 65 F F F 13 260-279 34 34 34 63 63 63 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 36 36 36 67 67 67 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 33 33 33 62 62 62 F F F 19 380-399 34 34 34 64 64 64 F F F 20 400-419 35 35 35 65 65 65 F F F 21 420-439 37 37 37 70 70 70 F F F 22 440-459 24 24 24 43 43 43 F F F 23 460-479 37 37 37 70 70 70 F F F 24 480-499 36 36 36 68 68 68 F F F 25 500-519 35 35 35 65 65 65 F F F 26 520-539 32 32 32 61 61 61 F F F 27 540-559 32 32 32 60 60 60 F F F 28 560-579 32 32 32 61 61 61 F F F 29 580-599 37 37 37 70 70 70 F F F Max gen: 37, Max flush: 70 Begin loop 19 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.89962435Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.907758381Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908042459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908072265Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908082554Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908093992Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286} {"msg":"Crucible 8aab1fdc-f7d8-41c2-adb6-5919a4d71f50 has session id: e4393407-5d9d-41b3-971c-4a6c2b66d30c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908389809Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Upstairs opts: Upstairs UUID: 8aab1fdc-f7d8-41c2-adb6-5919a4d71f50, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908459868Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Crucible stats registered with UUID: 8aab1fdc-f7d8-41c2-adb6-5919a4d71f50","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908487104Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:44.908497463Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908614716Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"io task","client":"2","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908637922Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"io task","client":"1","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908647951Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908776732Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"task":"crutest"} {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.90879017Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286} {"msg":"8aab1fdc-f7d8-41c2-adb6-5919a4d71f50 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908807298Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908816036Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"client":"0","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908825585Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"client":"1","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908835094Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"client":"2","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908846802Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"io task","client":"2","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908864129Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"io task","client":"1","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908874538Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"ds_connection connected from Ok(127.0.0.1:43769)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908936209Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"io task","client":"2","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"ds_connection connected from Ok(127.0.0.1:45750)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908949757Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"io task","client":"1","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"ds_connection connected from Ok(127.0.0.1:35286)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.908959945Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.909631887Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"client":"1","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.909659233Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"client":"0","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.909719754Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"client":"2","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.909967178Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910012611Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"[0]R flush_numbers[0..12]: [64, 59, 53, 63, 64, 68, 68, 66, 64, 69, 61, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910053895Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"[0]R generation[0..12]: [34, 32, 30, 34, 34, 36, 36, 36, 34, 36, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910065644Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910076622Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"[1]R flush_numbers[0..12]: [64, 59, 53, 63, 64, 68, 68, 66, 64, 69, 61, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910087081Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"[1]R generation[0..12]: [34, 32, 30, 34, 34, 36, 36, 36, 34, 36, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910099949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910110837Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"[2]R flush_numbers[0..12]: [64, 59, 53, 63, 64, 68, 68, 66, 64, 69, 61, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910121156Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"[2]R generation[0..12]: [34, 32, 30, 34, 34, 36, 36, 36, 34, 36, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910131314Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910142163Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Max found gen is 38","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910151481Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.91016113Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Next flush: 71","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910170558Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910180017Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"":"downstairs","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910189766Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910198424Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"client":"0","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910208083Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"client":"1","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910217522Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"client":"2","":"downstairs","session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.91022717Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910236169Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} Wait for a query_work_queue command to finish before sending IO {"msg":"8aab1fdc-f7d8-41c2-adb6-5919a4d71f50 is now active with session: e4393407-5d9d-41b3-971c-4a6c2b66d30c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910250907Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910260045Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910267494Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286,"session_id":"e4393407-5d9d-41b3-971c-4a6c2b66d30c"} {"msg":"Activated sub_volume 8aab1fdc-f7d8-41c2-adb6-5919a4d71f50","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:44.910276553Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 376 len 20480 data: 5 5 4 4 3 02/30 Read block 249 len 4096 03/30 Write block 590 len 20480 data: 6 6 6 5 4 04/30 Write block 438 len 24576 data: 3 3 4 6 8 7 05/30 Read block 383 len 12288 06/30 Flush 07/30 Write block 8 len 4096 data: 6 08/30 Read block 440 len 12288 09/30 Read block 16 len 20480 10/30 Read block 570 len 20480 11/30 Write block 592 len 12288 data: 7 6 5 12/30 Read block 395 len 36864 13/30 Read block 311 len 16384 14/30 Read block 556 len 12288 15/30 Read block 68 len 36864 16/30 Read block 253 len 40960 17/30 Write block 587 len 4096 data: 7 18/30 Read block 327 len 8192 19/30 Read block 275 len 4096 20/30 Flush 21/30 Write block 413 len 4096 data: 4 22/30 Write block 431 len 20480 data: 3 3 3 5 4 23/30 Read block 574 len 4096 24/30 Write block 444 len 36864 data: 7 8 7 6 6 5 3 2 2 25/30 Flush 26/30 Write block 37 len 40960 data: 4 5 5 5 3 2 2 2 5 5 27/30 Read block 466 len 12288 28/30 Read block 176 len 16384 29/30 Read block 209 len 4096 30/30 Write block 54 len 16384 data: 3 3 4 4 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Write 6 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Read 3 Done Done Sent false 1008 Acked Read 5 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Write 3 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Read 3 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Read 10 Done Done Sent false 1016 Acked Write 1 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Read 1 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Write 9 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 25 27 Done 29 29 5 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:44.919157835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1286} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 34 72 72 64 F F F <--- 1 020-039 32 32 32 59 59 59 T T F <--- 2 040-059 30 30 30 53 53 53 T T F <--- 3 060-079 34 34 34 63 63 63 F F F 4 080-099 34 34 34 64 64 64 F F F 5 100-119 36 36 36 68 68 68 F F F 6 120-139 36 36 36 68 68 68 F F F 7 140-159 36 36 36 66 66 66 F F F 8 160-179 34 34 34 64 64 64 F F F 9 180-199 36 36 36 69 69 69 F F F 10 200-219 32 32 32 61 61 61 F F F 11 220-239 36 36 36 66 66 66 F F F 12 240-259 35 35 35 65 65 65 F F F 13 260-279 34 34 34 63 63 63 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 36 36 36 67 67 67 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 38 38 33 71 71 62 F F T <--- 19 380-399 38 38 34 71 71 64 F F T <--- 20 400-419 38 38 35 73 73 65 F F F <--- 21 420-439 38 38 37 73 73 70 F F T <--- 22 440-459 38 38 24 73 73 43 F F T <--- 23 460-479 37 37 37 70 70 70 F F F 24 480-499 36 36 36 68 68 68 F F F 25 500-519 35 35 35 65 65 65 F F F 26 520-539 32 32 32 61 61 61 F F F 27 540-559 32 32 32 60 60 60 F F F 28 560-579 32 32 32 61 61 61 F F F 29 580-599 38 38 37 72 72 70 F F T <--- Max gen: 38, Max flush: 73 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 72 72 72 F F F 1 020-039 39 39 39 74 74 74 F F F 2 040-059 39 39 39 74 74 74 F F F 3 060-079 34 34 34 63 63 63 F F F 4 080-099 34 34 34 64 64 64 F F F 5 100-119 36 36 36 68 68 68 F F F 6 120-139 36 36 36 68 68 68 F F F 7 140-159 36 36 36 66 66 66 F F F 8 160-179 34 34 34 64 64 64 F F F 9 180-199 36 36 36 69 69 69 F F F 10 200-219 32 32 32 61 61 61 F F F 11 220-239 36 36 36 66 66 66 F F F 12 240-259 35 35 35 65 65 65 F F F 13 260-279 34 34 34 63 63 63 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 36 36 36 67 67 67 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 38 38 38 71 71 71 F F F 19 380-399 38 38 38 71 71 71 F F F 20 400-419 38 38 38 73 73 73 F F F 21 420-439 38 38 38 73 73 73 F F F 22 440-459 38 38 38 73 73 73 F F F 23 460-479 37 37 37 70 70 70 F F F 24 480-499 36 36 36 68 68 68 F F F 25 500-519 35 35 35 65 65 65 F F F 26 520-539 32 32 32 61 61 61 F F F 27 540-559 32 32 32 60 60 60 F F F 28 560-579 32 32 32 61 61 61 F F F 29 580-599 38 38 38 72 72 72 F F F Max gen: 39, Max flush: 74 Begin loop 20 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.159367447Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.167447346Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.167736564Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.167772639Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.167781658Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.167793026Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294} {"msg":"Crucible 6ed904d2-bfda-418b-8f1d-df7ff521d459 has session id: ac97aec7-7a09-44f6-990b-896852ee547c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168149644Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Upstairs opts: Upstairs UUID: 6ed904d2-bfda-418b-8f1d-df7ff521d459, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168179929Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Crucible stats registered with UUID: 6ed904d2-bfda-418b-8f1d-df7ff521d459","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168191208Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:49.168200566Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168356544Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168393188Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"io task","client":"2","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168405826Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"io task","client":"0","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168498603Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"task":"crutest"} {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168512331Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294} {"msg":"6ed904d2-bfda-418b-8f1d-df7ff521d459 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168535397Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168544686Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"client":"0","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168554395Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"client":"1","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168563993Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"client":"2","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168573512Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"io task","client":"2","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168591349Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"ds_connection connected from Ok(127.0.0.1:47827)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168612786Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"io task","client":"2","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168632043Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"io task","client":"0","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"ds_connection connected from Ok(127.0.0.1:57449)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.168712542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"ds_connection connected from Ok(127.0.0.1:50311)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.16872392Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"io task","client":"0","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169240124Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"client":"2","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169258852Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"client":"0","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169311754Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"client":"1","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169483499Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169559388Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"[0]R flush_numbers[0..12]: [72, 74, 74, 63, 64, 68, 68, 66, 64, 69, 61, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169617209Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"[0]R generation[0..12]: [38, 39, 39, 34, 34, 36, 36, 36, 34, 36, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169628188Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169639226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"[1]R flush_numbers[0..12]: [72, 74, 74, 63, 64, 68, 68, 66, 64, 69, 61, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169649485Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"[1]R generation[0..12]: [38, 39, 39, 34, 34, 36, 36, 36, 34, 36, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169659703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169670542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"[2]R flush_numbers[0..12]: [72, 74, 74, 63, 64, 68, 68, 66, 64, 69, 61, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169685899Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"[2]R generation[0..12]: [38, 39, 39, 34, 34, 36, 36, 36, 34, 36, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169696148Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169706886Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169716115Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169725713Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Next flush: 75","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169734932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169745931Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"":"downstairs","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169755439Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169764088Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"client":"0","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169773546Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"client":"1","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169783015Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","Wait for a query_work_queue command to finish before sending IO pid":1294,"client":"2","":"downstairs","session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169796613Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169805582Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"6ed904d2-bfda-418b-8f1d-df7ff521d459 is now active with session: ac97aec7-7a09-44f6-990b-896852ee547c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.16981541Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169824479Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169832278Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294,"session_id":"ac97aec7-7a09-44f6-990b-896852ee547c"} {"msg":"Activated sub_volume 6ed904d2-bfda-418b-8f1d-df7ff521d459","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:49.169841367Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 334 len 12288 data: 5 4 5 02/30 Read block 206 len 32768 03/30 Read block 123 len 32768 04/30 Write block 571 len 8192 data: 4 4 05/30 Read block 323 len 8192 06/30 Write block 553 len 40960 data: 5 6 6 5 5 3 3 4 4 4 07/30 Read block 231 len 24576 08/30 Write block 479 len 20480 data: 2 2 2 3 3 09/30 Read block 404 len 40960 10/30 Read block 335 len 4096 11/30 Flush 12/30 Write block 507 len 16384 data: 4 4 4 3 13/30 Read block 84 len 36864 14/30 Write block 98 len 4096 data: 4 15/30 Write block 398 len 20480 data: 3 2 2 2 3 16/30 Read block 427 len 40960 17/30 Read block 53 len 4096 18/30 Read block 349 len 4096 19/30 Read block 513 len 16384 20/30 Read block 192 len 4096 21/30 Flush 22/30 Write block 157 len 28672 data: 4 4 5 5 5 6 8 23/30 Write block 28 len 12288 data: 6 5 4 24/30 Read block 409 len 36864 25/30 Read block 564 len 16384 26/30 Write block 34 len 4096 data: 3 27/30 Write block 172 len 36864 data: 4 4 3 3 3 2 3 3 4 28/30 Write block 63 len 12288 data: 2 2 2 29/30 Read block 480 len 8192 30/30 Read block 307 len 20480 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Read 8 Done Sent Done false 1002 Acked Read 8 Done Sent Done false 1003 Acked Write 2 Done Sent Done false 1004 Acked Read 2 Done Sent Done false 1005 Acked Write 10 Done Sent Done false 1006 Acked Read 6 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 4 Done Sent Done false 1012 Acked Read 9 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Read 9 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Read 5 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:49.178732207Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1294} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 72 72 72 F F F 1 020-039 39 39 39 74 74 74 T F T <--- 2 040-059 39 39 39 74 74 74 F F F 3 060-079 34 34 34 63 63 63 T F T <--- 4 080-099 40 34 40 76 64 76 F F F <--- 5 100-119 36 36 36 68 68 68 F F F 6 120-139 36 36 36 68 68 68 F F F 7 140-159 36 36 36 66 66 66 T F T <--- 8 160-179 34 34 34 64 64 64 T F T <--- 9 180-199 36 36 36 69 69 69 T F T <--- 10 200-219 32 32 32 61 61 61 F F F 11 220-239 36 36 36 66 66 66 F F F 12 240-259 35 35 35 65 65 65 F F F 13 260-279 34 34 34 63 63 63 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 40 36 40 75 67 75 F F F <--- 17 340-359 25 25 25 44 44 44 F F F 18 360-379 38 38 38 71 71 71 F F F 19 380-399 40 38 40 76 71 76 F F F <--- 20 400-419 40 38 40 76 73 76 F F F <--- 21 420-439 38 38 38 73 73 73 F F F 22 440-459 38 38 38 73 73 73 F F F 23 460-479 40 37 40 75 70 75 F F F <--- 24 480-499 40 36 40 75 68 75 F F F <--- 25 500-519 40 35 40 76 65 76 F F F <--- 26 520-539 32 32 32 61 61 61 F F F 27 540-559 40 32 40 75 60 75 F F F <--- 28 560-579 40 32 40 75 61 75 F F F <--- 29 580-599 38 38 38 72 72 72 F F F Max gen: 40, Max flush: 76 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 72 72 72 F F F 1 020-039 41 41 41 77 77 77 F F F 2 040-059 39 39 39 74 74 74 F F F 3 060-079 41 41 41 77 77 77 F F F 4 080-099 40 40 40 76 76 76 F F F 5 100-119 36 36 36 68 68 68 F F F 6 120-139 36 36 36 68 68 68 F F F 7 140-159 41 41 41 77 77 77 F F F 8 160-179 41 41 41 77 77 77 F F F 9 180-199 41 41 41 77 77 77 F F F 10 200-219 32 32 32 61 61 61 F F F 11 220-239 36 36 36 66 66 66 F F F 12 240-259 35 35 35 65 65 65 F F F 13 260-279 34 34 34 63 63 63 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 40 40 40 75 75 75 F F F 17 340-359 25 25 25 44 44 44 F F F 18 360-379 38 38 38 71 71 71 F F F 19 380-399 40 40 40 76 76 76 F F F 20 400-419 40 40 40 76 76 76 F F F 21 420-439 38 38 38 73 73 73 F F F 22 440-459 38 38 38 73 73 73 F F F 23 460-479 40 40 40 75 75 75 F F F 24 480-499 40 40 40 75 75 75 F F F 25 500-519 40 40 40 76 76 76 F F F 26 520-539 32 32 32 61 61 61 F F F 27 540-559 40 40 40 75 75 75 F F F 28 560-579 40 40 40 75 75 75 F F F 29 580-599 38 38 38 72 72 72 F F F Max gen: 41, Max flush: 77 Begin loop 21 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.430961643Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.439097184Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.439385862Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.439412908Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.439421326Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.439432785Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302} {"msg":"Crucible 402634cc-a2a0-4c41-8aef-8aadfad4179b has session id: 50cd819a-4409-4fb2-aa1f-478037a2f630","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.439762916Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Upstairs opts: Upstairs UUID: 402634cc-a2a0-4c41-8aef-8aadfad4179b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.439815089Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Crucible stats registered with UUID: 402634cc-a2a0-4c41-8aef-8aadfad4179b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.439825507Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:53.439834406Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440016619Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440039406Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"io task","client":"2","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440049265Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"io task","client":"1","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440156619Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"task":"crutest"} {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440169827Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302} {"msg":"402634cc-a2a0-4c41-8aef-8aadfad4179b active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440207241Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.44021649Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"client":"0","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440226479Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"client":"1","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440235977Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"client":"2","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440245326Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"io task","client":"2","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440257794Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440268852Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"io task","client":"1","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"ds_connection connected from Ok(127.0.0.1:46378)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.44028702Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"io task","client":"2","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"ds_connection connected from Ok(127.0.0.1:48789)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440336962Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"io task","client":"1","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"ds_connection connected from Ok(127.0.0.1:52460)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440448926Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440884442Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"client":"2","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440905069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"client":"0","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.440999886Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"client":"1","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441169291Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441184769Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"[0]R flush_numbers[0..12]: [72, 77, 74, 77, 76, 68, 68, 77, 77, 77, 61, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441265617Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"[0]R generation[0..12]: [38, 41, 39, 41, 40, 36, 36, 41, 41, 41, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441276685Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441287704Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"[1]R flush_numbers[0..12]: [72, 77, 74, 77, 76, 68, 68, 77, 77, 77, 61, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441298002Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"[1]R generation[0..12]: [38, 41, 39, 41, 40, 36, 36, 41, 41, 41, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441308281Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441320809Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"[2]R flush_numbers[0..12]: [72, 77, 74, 77, 76, 68, 68, 77, 77, 77, 61, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441331187Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"[2]R generation[0..12]: [38, 41, 39, 41, 40, 36, 36, 41, 41, 41, 32, 36]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441341386Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441352124Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441361543Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441371111Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Next flush: 78","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.44138028Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441389769Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"":"downstairs","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441399427Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441408096Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"client":"0","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441417675Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"client":"1","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441427343Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"client":"2","":"downstairs","session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441437102Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Set Downstairs and Upstairs active after reconciliation","Wait for a query_work_queue command to finish before sending IO v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.44144987Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"402634cc-a2a0-4c41-8aef-8aadfad4179b is now active with session: 50cd819a-4409-4fb2-aa1f-478037a2f630","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441459638Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441470177Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441477986Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302,"session_id":"50cd819a-4409-4fb2-aa1f-478037a2f630"} {"msg":"Activated sub_volume 402634cc-a2a0-4c41-8aef-8aadfad4179b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:53.441487194Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 182 len 12288 data: 5 5 5 02/30 Read block 383 len 40960 03/30 Read block 47 len 40960 04/30 Read block 591 len 20480 05/30 Write block 39 len 32768 data: 6 6 4 3 3 3 6 6 06/30 Read block 3 len 32768 07/30 Write block 237 len 16384 data: 4 7 7 7 08/30 Flush 09/30 Write block 262 len 24576 data: 6 7 7 7 7 8 10/30 Read block 350 len 28672 11/30 Write block 341 len 16384 data: 6 6 5 5 12/30 Read block 110 len 20480 13/30 Read block 578 len 28672 14/30 Write block 226 len 32768 data: 7 6 7 6 6 3 3 2 15/30 Write block 570 len 12288 data: 4 5 5 16/30 Write block 422 len 24576 data: 6 5 4 4 4 4 17/30 Read block 397 len 16384 18/30 Flush 19/30 Write block 567 len 40960 data: 4 3 3 5 6 6 4 5 6 7 20/30 Read block 527 len 40960 21/30 Read block 392 len 28672 22/30 Write block 595 len 8192 data: 3 3 23/30 Write block 453 len 32768 data: 2 4 4 5 5 6 6 4 24/30 Write block 418 len 28672 data: 5 5 4 4 7 6 5 25/30 Read block 587 len 8192 26/30 Read block 35 len 32768 27/30 Read block 537 len 20480 28/30 Write block 194 len 28672 data: 4 4 4 5 6 6 5 29/30 Write block 35 len 36864 data: 3 4 5 6 7 7 5 4 4 30/30 Read block 355 len 8192 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Read 10 Done Sent Done false 1003 Acked Read 5 Done Sent Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Write 4 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Read 7 Done Sent Done false 1021 Acked Write 2 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Write 7 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Write 9 Sent Sent Done false 1029 Acked Read 2 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 28 0 30 Done 28 2 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) None Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:53.451113077Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1302} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 72 72 72 F F F 1 020-039 42 41 42 78 77 78 T F T <--- 2 040-059 42 39 42 78 74 78 T F T <--- 3 060-079 41 41 41 77 77 77 F F F 4 080-099 40 40 40 76 76 76 F F F 5 100-119 36 36 36 68 68 68 F F F 6 120-139 36 36 36 68 68 68 F F F 7 140-159 41 41 41 77 77 77 F F F 8 160-179 41 41 41 77 77 77 F F F 9 180-199 42 41 42 78 77 78 T T T <--- 10 200-219 32 32 32 61 61 61 T F T <--- 11 220-239 42 36 42 79 66 79 F F F <--- 12 240-259 42 35 42 78 65 78 F F F <--- 13 260-279 42 34 42 79 63 79 F F F <--- 14 280-299 32 32 32 61 61 61 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 40 40 40 75 75 75 F F F 17 340-359 42 25 42 79 44 79 F F F <--- 18 360-379 38 38 38 71 71 71 F F F 19 380-399 40 40 40 76 76 76 F F F 20 400-419 40 40 40 76 76 76 T F T <--- 21 420-439 42 38 42 79 73 79 T F T <--- 22 440-459 38 38 38 73 73 73 T F T <--- 23 460-479 40 40 40 75 75 75 T F T <--- 24 480-499 40 40 40 75 75 75 F F F 25 500-519 40 40 40 76 76 76 F F F 26 520-539 32 32 32 61 61 61 F F F 27 540-559 40 40 40 75 75 75 F F F 28 560-579 42 40 42 79 75 79 T F T <--- 29 580-599 38 38 38 72 72 72 T F T <--- Max gen: 42, Max flush: 79 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 72 72 72 F F F 1 020-039 43 43 43 80 80 80 F F F 2 040-059 43 43 43 80 80 80 F F F 3 060-079 41 41 41 77 77 77 F F F 4 080-099 40 40 40 76 76 76 F F F 5 100-119 36 36 36 68 68 68 F F F 6 120-139 36 36 36 68 68 68 F F F 7 140-159 41 41 41 77 77 77 F F F 8 160-179 41 41 41 77 77 77 F F F 9 180-199 43 43 43 80 80 80 F F F 10 200-219 43 43 43 80 80 80 F F F 11 220-239 42 42 42 79 79 79 F F F 12 240-259 42 42 42 78 78 78 F F F 13 260-279 42 42 42 79 79 79 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 40 40 40 75 75 75 F F F 17 340-359 42 42 42 79 79 79 F F F 18 360-379 38 38 38 71 71 71 F F F 19 380-399 40 40 40 76 76 76 F F F 20 400-419 43 43 43 80 80 80 F F F 21 420-439 43 43 43 80 80 80 F F F 22 440-459 43 43 43 80 80 80 F F F 23 460-479 43 43 43 80 80 80 F F F 24 480-499 40 40 40 75 75 75 F F F 25 500-519 40 40 40 76 76 76 F F F 26 520-539 32 32 32 61 61 61 F F F 27 540-559 40 40 40 75 75 75 F F F 28 560-579 43 43 43 80 80 80 F F F 29 580-599 43 43 43 80 80 80 F F F Max gen: 43, Max flush: 80 Begin loop 22 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.704268818Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.712637565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.712960257Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.712987173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.712995602Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713007191Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310} {"msg":"Crucible f123a0a1-35f7-4840-bd02-ec4815fd86c3 has session id: 972c8280-6b45-43ba-bbcb-ce56b3ace5e2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713325164Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Upstairs opts: Upstairs UUID: f123a0a1-35f7-4840-bd02-ec4815fd86c3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.71335349Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Crucible stats registered with UUID: f123a0a1-35f7-4840-bd02-ec4815fd86c3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713364938Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:57.713374307Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713516886Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"io task","client":"1","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713539053Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713550851Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713698989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"task":"crutest"} {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713716637Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310} {"msg":"f123a0a1-35f7-4840-bd02-ec4815fd86c3 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713756381Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.71376573Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"client":"0","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713775368Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"client":"1","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713784927Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"client":"2","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713794485Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713804454Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713814203Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"io task","client":"1","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"ds_connection connected from Ok(127.0.0.1:42610)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713881603Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"io task","client":"1","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"ds_connection connected from Ok(127.0.0.1:61308)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713907309Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"ds_connection connected from Ok(127.0.0.1:42143)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.713918547Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714528698Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"client":"0","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714577671Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"client":"1","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714600738Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"client":"2","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714877737Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714887556Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"[0]R flush_numbers[0..12]: [72, 80, 80, 77, 76, 68, 68, 77, 77, 80, 80, 79]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714912372Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"[0]R generation[0..12]: [38, 43, 43, 41, 40, 36, 36, 41, 41, 43, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.71492322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714934289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"[1]R flush_numbers[0..12]: [72, 80, 80, 77, 76, 68, 68, 77, 77, 80, 80, 79]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714944567Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"[1]R generation[0..12]: [38, 43, 43, 41, 40, 36, 36, 41, 41, 43, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714954916Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714965734Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"[2]R flush_numbers[0..12]: [72, 80, 80, 77, 76, 68, 68, 77, 77, 80, 80, 79]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714976033Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"[2]R generation[0..12]: [38, 43, 43, 41, 40, 36, 36, 41, 41, 43, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.714986191Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.71499686Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Max found gen is 44","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715006248Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715015727Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Next flush: 81","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715024816Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715035484Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"":"downstairs","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715045423Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715054191Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"client":"0","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.71506401Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"client":"1","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Transition from WaitQuorum to Active","v":0,"Wait for a query_work_queue command to finish before sending IO name":"crucible","level":30,"time":"2026-04-15T18:11:57.715078228Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"client":"2","":"downstairs","session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715088066Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715096975Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"f123a0a1-35f7-4840-bd02-ec4815fd86c3 is now active with session: 972c8280-6b45-43ba-bbcb-ce56b3ace5e2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715106734Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715115322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310,"session_id":"972c8280-6b45-43ba-bbcb-ce56b3ace5e2"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.715124201Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310} {"msg":"Activated sub_volume f123a0a1-35f7-4840-bd02-ec4815fd86c3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:11:57.71513203Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 174 len 36864 data: 4 4 4 3 4 4 5 5 6 02/30 Read block 194 len 40960 03/30 Read block 351 len 16384 04/30 Write block 517 len 16384 data: 5 4 4 4 05/30 Write block 263 len 16384 data: 8 8 8 8 06/30 Write block 577 len 8192 data: 5 5 07/30 Flush 08/30 Write block 105 len 20480 data: 5 6 7 6 5 09/30 Read block 434 len 4096 10/30 Write block 183 len 4096 data: 6 11/30 Write block 324 len 8192 data: 5 5 12/30 Read block 249 len 24576 13/30 Flush 14/30 Write block 95 len 36864 data: 3 4 5 5 5 5 6 5 4 15/30 Write block 309 len 28672 data: 3 4 4 4 4 4 4 16/30 Flush 17/30 Write block 348 len 8192 data: 5 6 18/30 Write block 28 len 16384 data: 7 6 5 3 19/30 Read block 387 len 20480 20/30 Read block 181 len 32768 21/30 Flush 22/30 Flush 23/30 Write block 116 len 28672 data: 5 5 5 5 7 6 6 24/30 Read block 341 len 32768 25/30 Read block 74 len 36864 26/30 Write block 262 len 28672 data: 7 9 9 9 9 9 8 27/30 Read block 388 len 36864 28/30 Write block 446 len 20480 data: 8 7 7 6 4 29/30 Write block 501 len 12288 data: 2 2 2 30/30 Read block 568 len 16384 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Read 4 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Read 1 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Write 3 Done Sent Sent false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 2 30 Done 30 2 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:11:57.724076233Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1310} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 72 72 72 F F F 1 020-039 44 43 44 84 80 84 F F F <--- 2 040-059 43 43 43 80 80 80 F F F 3 060-079 41 41 41 77 77 77 F F F 4 080-099 44 40 44 83 76 83 F F F <--- 5 100-119 44 36 44 83 68 83 T F T <--- 6 120-139 36 36 36 68 68 68 T F T <--- 7 140-159 41 41 41 77 77 77 F F F 8 160-179 44 41 44 81 77 81 F T F <--- 9 180-199 44 43 44 82 80 82 F T F <--- 10 200-219 43 43 43 80 80 80 F F F 11 220-239 42 42 42 79 79 79 F F F 12 240-259 42 42 42 78 78 78 F F F 13 260-279 44 42 44 81 79 81 T F T <--- 14 280-299 32 32 32 61 61 61 F F F 15 300-319 44 36 44 83 67 83 F F F <--- 16 320-339 44 40 44 82 75 82 F F F <--- 17 340-359 44 42 44 84 79 84 F F F <--- 18 360-379 38 38 38 71 71 71 F F F 19 380-399 40 40 40 76 76 76 F F F 20 400-419 43 43 43 80 80 80 F F F 21 420-439 43 43 43 80 80 80 F F F 22 440-459 43 43 43 80 80 80 T F T <--- 23 460-479 43 43 43 80 80 80 F F F 24 480-499 40 40 40 75 75 75 F F F 25 500-519 44 40 44 81 76 81 T F T <--- 26 520-539 44 32 44 81 61 81 F F F <--- 27 540-559 40 40 40 75 75 75 F F F 28 560-579 44 43 44 81 80 81 F F F <--- 29 580-599 43 43 43 80 80 80 F F F Max gen: 44, Max flush: 84 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 22 Verifying data now No change after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 72 72 72 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 43 43 43 80 80 80 F F F 3 060-079 41 41 41 77 77 77 F F F 4 080-099 44 44 44 83 83 83 F F F 5 100-119 45 45 45 85 85 85 F F F 6 120-139 45 45 45 85 85 85 F F F 7 140-159 41 41 41 77 77 77 F F F 8 160-179 44 44 44 81 81 81 F F F 9 180-199 44 44 44 82 82 82 F F F 10 200-219 43 43 43 80 80 80 F F F 11 220-239 42 42 42 79 79 79 F F F 12 240-259 42 42 42 78 78 78 F F F 13 260-279 45 45 45 85 85 85 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 44 44 44 83 83 83 F F F 16 320-339 44 44 44 82 82 82 F F F 17 340-359 44 44 44 84 84 84 F F F 18 360-379 38 38 38 71 71 71 F F F 19 380-399 40 40 40 76 76 76 F F F 20 400-419 43 43 43 80 80 80 F F F 21 420-439 43 43 43 80 80 80 F F F 22 440-459 45 45 45 85 85 85 F F F 23 460-479 43 43 43 80 80 80 F F F 24 480-499 40 40 40 75 75 75 F F F 25 500-519 45 45 45 85 85 85 F F F 26 520-539 44 44 44 81 81 81 F F F 27 540-559 40 40 40 75 75 75 F F F 28 560-579 44 44 44 81 81 81 F F F 29 580-599 43 43 43 80 80 80 F F F Max gen: 45, Max flush: 85 Begin loop 23 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.976305808Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:01.983506285Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.983570186Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.990649331Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.990924961Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.990952707Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.990963256Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.990974594Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318} {"msg":"Crucible 14b8da8e-0214-4a3a-9f71-5360a0db259e has session id: 4db26a7c-e85e-40f9-9144-f4b73aff73ff","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991366967Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Upstairs opts: Upstairs UUID: 14b8da8e-0214-4a3a-9f71-5360a0db259e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991390603Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Crucible stats registered with UUID: 14b8da8e-0214-4a3a-9f71-5360a0db259e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991400482Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:01.99140952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991582925Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"io task","client":"1","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991622059Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"io task","client":"0","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991633028Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991701148Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"task":"crutest"} {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991721945Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318} {"msg":"14b8da8e-0214-4a3a-9f71-5360a0db259e active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.99175691Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991766378Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"client":"0","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991776047Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"client":"1","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991785625Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"client":"2","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991797124Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991836368Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"io task","client":"0","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:55194)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991846786Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.991990775Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"io task","client":"1","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:33998)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992006873Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"io task","client":"0","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:60972)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992017811Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"io task","client":"1","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992450928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"client":"2","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992466336Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"client":"1","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992564661Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"client":"0","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992760353Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992790678Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"[0]R flush_numbers[0..12]: [72, 84, 80, 77, 83, 85, 85, 77, 81, 82, 80, 79]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992859708Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"[0]R generation[0..12]: [38, 44, 43, 41, 44, 45, 45, 41, 44, 44, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992871527Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992882505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"[1]R flush_numbers[0..12]: [72, 84, 80, 77, 83, 85, 85, 77, 81, 82, 80, 79]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992892983Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"[1]R generation[0..12]: [38, 44, 43, 41, 44, 45, 45, 41, 44, 44, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992906551Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.99291758Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"[2]R flush_numbers[0..12]: [72, 84, 80, 77, 83, 85, 85, 77, 81, 82, 80, 79]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992928178Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"[2]R generation[0..12]: [38, 44, 43, 41, 44, 45, 45, 41, 44, 44, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992938447Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992949245Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992958624Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992968262Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Next flush: 86","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992977561Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.99298683Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"":"downstairs","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.992996418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.993016255Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"client":"0","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:12:01.993025994Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"client":"1","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.993037872Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"client":"2","":"downstairs","session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.993047511Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.9930566Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"14b8da8e-0214-4a3a-9f71-5360a0db259e is now active with session: 4db26a7c-e85e-40f9-9144-f4b73aff73ff","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.993071407Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.993080466Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.993087845Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318,"session_id":"4db26a7c-e85e-40f9-9144-f4b73aff73ff"} {"msg":"Activated sub_volume 14b8da8e-0214-4a3a-9f71-5360a0db259e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:01.993096854Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 313 len 16384 data: 5 5 5 8 03/30 Read block 488 len 40960 04/30 Write block 78 len 20480 data: 5 4 4 5 6 05/30 Write block 190 len 16384 data: 3 4 4 4 06/30 Write block 345 len 8192 data: 7 7 07/30 Write block 511 len 20480 data: 3 3 4 6 4 08/30 Write block 204 len 24576 data: 4 4 4 3 3 3 09/30 Write block 98 len 16384 data: 6 6 6 7 10/30 Write block 130 len 20480 data: 3 2 3 5 4 11/30 Write block 445 len 20480 data: 9 9 8 8 7 12/30 Read block 409 len 8192 13/30 Read block 202 len 20480 14/30 Read block 284 len 24576 15/30 Write block 433 len 24576 data: 4 6 5 4 3 4 16/30 Write block 109 len 12288 data: 6 5 5 17/30 Read block 331 len 40960 18/30 Write block 169 len 36864 data: 5 4 4 5 5 5 5 5 4 19/30 Read block 522 len 12288 20/30 Read block 64 len 4096 21/30 Flush 22/30 Write block 456 len 12288 data: 6 6 7 23/30 Read block 322 len 24576 24/30 Read block 91 len 24576 25/30 Read block 63 len 16384 26/30 Read block 301 len 24576 27/30 Read block 524 len 4096 28/30 Write block 227 len 12288 data: 7 8 7 29/30 Write block 178 len 12288 data: 5 5 6 30/30 Write block 61 len 24576 data: 3 2 3 3 3 2 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Sent Done Done false 1002 Acked Read 10 Sent Done Done false 1003 Acked Write 5 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Write 3 Sent Done Done false 1016 Acked Read 10 Sent Done Done false 1017 Acked Write 9 Sent Done Done false 1018 Acked Read 3 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 3 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 3 3 35 Done 0 26 26 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:02.00126076Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1318} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 72 72 72 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 43 43 43 80 80 80 F F F 3 060-079 41 46 46 77 87 87 F T T <--- 4 080-099 44 46 46 83 87 87 F F F <--- 5 100-119 45 46 46 85 87 87 F F F <--- 6 120-139 45 46 46 85 87 87 F F F <--- 7 140-159 41 41 41 77 77 77 F F F 8 160-179 44 46 46 81 87 87 F T T <--- 9 180-199 44 46 46 82 87 87 F T T <--- 10 200-219 43 46 46 80 87 87 F F F <--- 11 220-239 42 42 42 79 79 79 F T T <--- 12 240-259 42 42 42 78 78 78 F F F 13 260-279 45 45 45 85 85 85 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 44 46 46 83 87 87 F F F <--- 16 320-339 44 44 44 82 82 82 F F F 17 340-359 44 46 46 84 87 87 F F F <--- 18 360-379 38 38 38 71 71 71 F F F 19 380-399 40 40 40 76 76 76 F F F 20 400-419 43 43 43 80 80 80 F F F 21 420-439 43 46 46 80 87 87 F F F <--- 22 440-459 45 46 46 85 87 87 F T T <--- 23 460-479 43 43 43 80 80 80 F F F 24 480-499 40 40 40 75 75 75 F F F 25 500-519 45 46 46 85 87 87 F F F <--- 26 520-539 44 44 44 81 81 81 F F F 27 540-559 40 40 40 75 75 75 F F F 28 560-579 44 44 44 81 81 81 F F F 29 580-599 43 43 43 80 80 80 F F F Max gen: 46, Max flush: 87 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 72 72 72 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 43 43 43 80 80 80 F F F 3 060-079 47 47 47 88 88 88 F F F 4 080-099 46 46 46 87 87 87 F F F 5 100-119 46 46 46 87 87 87 F F F 6 120-139 46 46 46 87 87 87 F F F 7 140-159 41 41 41 77 77 77 F F F 8 160-179 47 47 47 88 88 88 F F F 9 180-199 47 47 47 88 88 88 F F F 10 200-219 46 46 46 87 87 87 F F F 11 220-239 47 47 47 88 88 88 F F F 12 240-259 42 42 42 78 78 78 F F F 13 260-279 45 45 45 85 85 85 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 46 46 46 87 87 87 F F F 16 320-339 44 44 44 82 82 82 F F F 17 340-359 46 46 46 87 87 87 F F F 18 360-379 38 38 38 71 71 71 F F F 19 380-399 40 40 40 76 76 76 F F F 20 400-419 43 43 43 80 80 80 F F F 21 420-439 46 46 46 87 87 87 F F F 22 440-459 47 47 47 88 88 88 F F F 23 460-479 43 43 43 80 80 80 F F F 24 480-499 40 40 40 75 75 75 F F F 25 500-519 46 46 46 87 87 87 F F F 26 520-539 44 44 44 81 81 81 F F F 27 540-559 40 40 40 75 75 75 F F F 28 560-579 44 44 44 81 81 81 F F F 29 580-599 43 43 43 80 80 80 F F F Max gen: 47, Max flush: 88 Begin loop 24 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.257070673Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:06.264238095Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.264300976Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.271421655Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.271682317Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.271710343Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.271718832Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.27173041Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326} {"msg":"Crucible 020b5870-763f-48ce-bad3-a687b62c42aa has session id: 11c2d841-6228-411d-99c0-f4313981a553","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272045744Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Upstairs opts: Upstairs UUID: 020b5870-763f-48ce-bad3-a687b62c42aa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272085918Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Crucible stats registered with UUID: 020b5870-763f-48ce-bad3-a687b62c42aa","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272097437Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:06.272108765Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272253414Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272275351Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"io task","client":"0","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272285239Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"io task","client":"1","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272423559Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"task":"crutest"} {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272441196Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326} {"msg":"020b5870-763f-48ce-bad3-a687b62c42aa active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272464403Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272473772Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"client":"0","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.27248344Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"client":"1","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272493249Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"client":"2","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272502827Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272512776Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"io task","client":"0","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272522754Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"io task","client":"1","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"ds_connection connected from Ok(127.0.0.1:65025)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272609282Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"ds_connection connected from Ok(127.0.0.1:44154)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272649936Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"io task","client":"0","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"ds_connection connected from Ok(127.0.0.1:48613)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.272672133Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"io task","client":"1","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273342565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"client":"2","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273365681Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"client":"1","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273385478Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"client":"0","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273653549Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273687324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"[0]R flush_numbers[0..12]: [72, 84, 80, 88, 87, 87, 87, 77, 88, 88, 87, 88]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273732548Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"[0]R generation[0..12]: [38, 44, 43, 47, 46, 46, 46, 41, 47, 47, 46, 47]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273743506Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273754374Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"[1]R flush_numbers[0..12]: [72, 84, 80, 88, 87, 87, 87, 77, 88, 88, 87, 88]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273764653Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"[1]R generation[0..12]: [38, 44, 43, 47, 46, 46, 46, 41, 47, 47, 46, 47]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273774951Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.2737858Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"[2]R flush_numbers[0..12]: [72, 84, 80, 88, 87, 87, 87, 77, 88, 88, 87, 88]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273800468Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"[2]R generation[0..12]: [38, 44, 43, 47, 46, 46, 46, 41, 47, 47, 46, 47]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273810756Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273821605Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273833633Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273843461Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Next flush: 89","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.27385261Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273862169Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"":"downstairs","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273871777Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273880576Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"client":"0","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273890264Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"client":"1","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273899773Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"client":"2","":"downstairs","session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273909562Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326Wait for a query_work_queue command to finish before sending IO ,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.27392252Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"020b5870-763f-48ce-bad3-a687b62c42aa is now active with session: 11c2d841-6228-411d-99c0-f4313981a553","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273932408Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273941507Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273949306Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326,"session_id":"11c2d841-6228-411d-99c0-f4313981a553"} {"msg":"Activated sub_volume 020b5870-763f-48ce-bad3-a687b62c42aa","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:06.273958375Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 11 len 24576 data: 6 6 5 4 4 4 02/30 Read block 326 len 36864 03/30 Write block 435 len 32768 data: 6 5 4 5 4 5 7 9 04/30 Write block 505 len 8192 data: 3 4 05/30 Write block 332 len 4096 data: 6 06/30 Read block 288 len 40960 07/30 Write block 336 len 32768 data: 6 4 5 5 6 7 7 6 08/30 Write block 342 len 32768 data: 8 7 6 8 8 6 6 7 09/30 Write block 138 len 4096 data: 3 10/30 Read block 118 len 36864 11/30 Read block 550 len 20480 12/30 Write block 48 len 20480 data: 5 4 3 3 3 13/30 Write block 212 len 36864 data: 3 3 3 3 4 4 5 5 3 14/30 Read block 452 len 12288 15/30 Read block 504 len 32768 16/30 Read block 393 len 12288 17/30 Read block 546 len 8192 18/30 Write block 204 len 4096 data: 5 19/30 Read block 165 len 12288 20/30 Read block 567 len 8192 21/30 Read block 262 len 12288 22/30 Write block 426 len 36864 data: 5 5 4 4 3 4 4 5 7 23/30 Read block 236 len 32768 24/30 Write block 394 len 40960 data: 3 3 3 3 4 3 3 3 4 3 25/30 Write block 92 len 24576 data: 3 2 3 4 5 6 26/30 Write block 139 len 28672 data: 6 8 9 8 9 9 10 27/30 Read block 338 len 40960 28/30 Read block 360 len 32768 29/30 Write block 404 len 40960 data: 4 4 6 5 5 4 6 6 5 5 30/30 Write block 47 len 4096 data: 5 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Write 2 Done Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Read 10 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Read 3 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 7 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 2 2 30 Done 4 28 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:06.282565087Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1326} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 72 72 72 T T T 1 020-039 44 44 44 84 84 84 F F F 2 040-059 43 43 43 80 80 80 F T T <--- 3 060-079 47 47 47 88 88 88 F F F 4 080-099 46 46 46 87 87 87 F T T <--- 5 100-119 46 46 46 87 87 87 F F F 6 120-139 46 46 46 87 87 87 F T T <--- 7 140-159 41 41 41 77 77 77 F T T <--- 8 160-179 47 47 47 88 88 88 F F F 9 180-199 47 47 47 88 88 88 F F F 10 200-219 46 46 46 87 87 87 F T T <--- 11 220-239 47 47 47 88 88 88 F T T <--- 12 240-259 42 42 42 78 78 78 F F F 13 260-279 45 45 45 85 85 85 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 46 46 46 87 87 87 F F F 16 320-339 44 44 44 82 82 82 F T T <--- 17 340-359 46 46 46 87 87 87 F T T <--- 18 360-379 38 38 38 71 71 71 F F F 19 380-399 40 40 40 76 76 76 F T T <--- 20 400-419 43 43 43 80 80 80 F T T <--- 21 420-439 46 46 46 87 87 87 T T T 22 440-459 47 47 47 88 88 88 T T T 23 460-479 43 43 43 80 80 80 F F F 24 480-499 40 40 40 75 75 75 F F F 25 500-519 46 46 46 87 87 87 T T T 26 520-539 44 44 44 81 81 81 F F F 27 540-559 40 40 40 75 75 75 F F F 28 560-579 44 44 44 81 81 81 F F F 29 580-599 43 43 43 80 80 80 F F F Max gen: 47, Max flush: 88 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 24 Verifying data now Files /tmp/test_live_repair/test_repair_verify.data and /tmp/test_live_repair/test_repair_verify.data.last differ 428a429,430 > 5, > 5, 432,435d433 < 3, < 2, < 3, < 3, 436a435,437 > 4, > 5, > 7, 438d438 < 6, diff found after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 89 89 89 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 48 48 48 89 89 89 F F F 3 060-079 47 47 47 88 88 88 F F F 4 080-099 48 48 48 89 89 89 F F F 5 100-119 46 46 46 87 87 87 F F F 6 120-139 48 48 48 89 89 89 F F F 7 140-159 48 48 48 89 89 89 F F F 8 160-179 47 47 47 88 88 88 F F F 9 180-199 47 47 47 88 88 88 F F F 10 200-219 48 48 48 89 89 89 F F F 11 220-239 48 48 48 89 89 89 F F F 12 240-259 42 42 42 78 78 78 F F F 13 260-279 45 45 45 85 85 85 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 46 46 46 87 87 87 F F F 16 320-339 48 48 48 89 89 89 F F F 17 340-359 48 48 48 89 89 89 F F F 18 360-379 38 38 38 71 71 71 F F F 19 380-399 48 48 48 89 89 89 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 48 48 48 89 89 89 F F F 22 440-459 48 48 48 89 89 89 F F F 23 460-479 43 43 43 80 80 80 F F F 24 480-499 40 40 40 75 75 75 F F F 25 500-519 48 48 48 89 89 89 F F F 26 520-539 44 44 44 81 81 81 F F F 27 540-559 40 40 40 75 75 75 F F F 28 560-579 44 44 44 81 81 81 F F F 29 580-599 43 43 43 80 80 80 F F F Max gen: 48, Max flush: 89 Begin loop 25 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.553726315Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.562087223Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.562405337Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.562431103Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.562439522Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.56245137Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337} {"msg":"Crucible 6fc46f91-c5b3-4af7-95c8-a9f004a86b83 has session id: a840e192-35be-46ad-9df5-d68217c62923","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.562819176Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Upstairs opts: Upstairs UUID: 6fc46f91-c5b3-4af7-95c8-a9f004a86b83, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.562841323Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Crucible stats registered with UUID: 6fc46f91-c5b3-4af7-95c8-a9f004a86b83","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.562851172Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:10.56286009Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563035115Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"io task","client":"1","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563072809Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"io task","client":"0","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563083688Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"io task","client":"2","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563189932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"task":"crutest"} {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.56320686Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337} {"msg":"6fc46f91-c5b3-4af7-95c8-a9f004a86b83 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563242124Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563251623Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"client":"0","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563263421Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"client":"1","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.56327316Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"client":"2","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563282568Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"io task","client":"2","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563292447Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"io task","client":"1","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563345069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"io task","client":"0","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"ds_connection connected from Ok(127.0.0.1:50681)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563371116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"io task","client":"1","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"ds_connection connected from Ok(127.0.0.1:50562)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563382634Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"io task","client":"2","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"ds_connection connected from Ok(127.0.0.1:53908)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563515334Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"io task","client":"0","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.563994894Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"client":"0","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564037418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"client":"1","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564114407Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"client":"2","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564260626Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564277113Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"[0]R flush_numbers[0..12]: [89, 84, 89, 88, 89, 87, 89, 89, 88, 88, 89, 89]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564372929Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"[0]R generation[0..12]: [48, 44, 48, 47, 48, 46, 48, 48, 47, 47, 48, 48]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564383937Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564397376Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"[1]R flush_numbers[0..12]: [89, 84, 89, 88, 89, 87, 89, 89, 88, 88, 89, 89]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564407844Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"[1]R generation[0..12]: [48, 44, 48, 47, 48, 46, 48, 48, 47, 47, 48, 48]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564418162Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564428931Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"[2]R flush_numbers[0..12]: [89, 84, 89, 88, 89, 87, 89, 89, 88, 88, 89, 89]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564439149Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"[2]R generation[0..12]: [48, 44, 48, 47, 48, 46, 48, 48, 47, 47, 48, 48]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564449358Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564460206Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Max found gen is 49","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564469595Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Generation requested: 50 >= found:49","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564479244Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Next flush: 90","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564488572Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564497881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"":"downstairs","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564507479Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564516208Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"client":"0","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564525867Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"client":"1","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564536705Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"client":"2","":"downstairs","session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564546414Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564555372Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} Wait for a query_work_queue command to finish before sending IO {"msg":"6fc46f91-c5b3-4af7-95c8-a9f004a86b83 is now active with session: a840e192-35be-46ad-9df5-d68217c62923","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.56456925Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564578529Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564586378Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337,"session_id":"a840e192-35be-46ad-9df5-d68217c62923"} {"msg":"Activated sub_volume 6fc46f91-c5b3-4af7-95c8-a9f004a86b83","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:10.564595487Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 226 len 20480 data: 8 8 9 8 7 02/30 Read block 404 len 24576 03/30 Flush 04/30 Write block 459 len 36864 data: 7 5 4 4 5 3 3 6 6 05/30 Flush 06/30 Write block 255 len 20480 data: 2 2 2 2 3 07/30 Read block 561 len 12288 08/30 Read block 517 len 8192 09/30 Write block 88 len 40960 data: 5 5 4 4 4 3 4 5 6 7 10/30 Flush 11/30 Write block 451 len 40960 data: 3 3 3 5 5 7 7 8 8 6 12/30 Read block 519 len 16384 13/30 Read block 38 len 40960 14/30 Write block 494 len 36864 data: 5 6 5 6 5 4 3 3 3 15/30 Write block 449 len 40960 data: 8 5 4 4 4 6 6 8 8 9 16/30 Read block 437 len 8192 17/30 Write block 313 len 36864 data: 6 6 6 9 8 8 7 8 6 18/30 Read block 130 len 20480 19/30 Write block 448 len 12288 data: 9 9 6 20/30 Write block 116 len 32768 data: 6 6 6 6 8 7 7 5 21/30 Read block 102 len 12288 22/30 Read block 225 len 20480 23/30 Flush 24/30 Write block 499 len 16384 data: 5 4 4 4 25/30 Write block 474 len 40960 data: 5 3 2 2 2 3 3 3 4 4 26/30 Read block 275 len 8192 27/30 Write block 558 len 28672 data: 4 4 5 5 5 4 4 28/30 Write block 373 len 36864 data: 5 7 5 6 6 5 5 4 3 29/30 Read block 402 len 16384 30/30 Write block 87 len 36864 data: 7 6 6 5 5 5 4 5 6 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Sent false 1001 Acked Read 6 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 9 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Read 3 Done Done Sent false 1007 Acked Read 2 Done Done Sent false 1008 Acked Write 10 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Read 4 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 5 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:10.575482755Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1337} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 89 89 89 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 48 48 48 89 89 89 F F F 3 060-079 47 47 47 88 88 88 F F F 4 080-099 50 50 48 92 92 89 T T F <--- 5 100-119 50 50 46 93 93 87 F F F <--- 6 120-139 50 50 48 93 93 89 F F F <--- 7 140-159 48 48 48 89 89 89 F F F 8 160-179 47 47 47 88 88 88 F F F 9 180-199 47 47 47 88 88 88 F F F 10 200-219 48 48 48 89 89 89 F F F 11 220-239 50 50 48 90 90 89 F F F <--- 12 240-259 50 50 42 92 92 78 F F F <--- 13 260-279 45 45 45 85 85 85 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 50 50 46 93 93 87 F F F <--- 16 320-339 50 50 48 93 93 89 F F F <--- 17 340-359 48 48 48 89 89 89 F F F 18 360-379 38 38 38 71 71 71 T T F <--- 19 380-399 48 48 48 89 89 89 T T F <--- 20 400-419 48 48 48 89 89 89 F F F 21 420-439 48 48 48 89 89 89 F F F 22 440-459 50 50 48 93 93 89 F F F <--- 23 460-479 50 50 43 93 93 80 T T F <--- 24 480-499 50 50 40 93 93 75 T T F <--- 25 500-519 50 50 48 93 93 89 T T F <--- 26 520-539 44 44 44 81 81 81 F F F 27 540-559 40 40 40 75 75 75 T T F <--- 28 560-579 44 44 44 81 81 81 T T F <--- 29 580-599 43 43 43 80 80 80 F F F Max gen: 50, Max flush: 93 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 25 Verifying data now No change after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 89 89 89 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 48 48 48 89 89 89 F F F 3 060-079 47 47 47 88 88 88 F F F 4 080-099 51 51 51 94 94 94 F F F 5 100-119 50 50 50 93 93 93 F F F 6 120-139 50 50 50 93 93 93 F F F 7 140-159 48 48 48 89 89 89 F F F 8 160-179 47 47 47 88 88 88 F F F 9 180-199 47 47 47 88 88 88 F F F 10 200-219 48 48 48 89 89 89 F F F 11 220-239 50 50 50 90 90 90 F F F 12 240-259 50 50 50 92 92 92 F F F 13 260-279 45 45 45 85 85 85 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 50 50 50 93 93 93 F F F 16 320-339 50 50 50 93 93 93 F F F 17 340-359 48 48 48 89 89 89 F F F 18 360-379 51 51 51 94 94 94 F F F 19 380-399 51 51 51 94 94 94 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 48 48 48 89 89 89 F F F 22 440-459 50 50 50 93 93 93 F F F 23 460-479 51 51 51 94 94 94 F F F 24 480-499 51 51 51 94 94 94 F F F 25 500-519 51 51 51 94 94 94 F F F 26 520-539 44 44 44 81 81 81 F F F 27 540-559 51 51 51 94 94 94 F F F 28 560-579 51 51 51 94 94 94 F F F 29 580-599 43 43 43 80 80 80 F F F Max gen: 51, Max flush: 94 Begin loop 26 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.830238721Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:14.837438169Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.837505559Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.844550989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.844825379Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.844852225Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.844860694Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.844872202Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345} {"msg":"Crucible d6631a6f-c777-4079-816d-a25bd634b5be has session id: 391bedd0-3978-4299-950f-d1d4a90eac71","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845167499Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Upstairs opts: Upstairs UUID: d6631a6f-c777-4079-816d-a25bd634b5be, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845191486Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Crucible stats registered with UUID: d6631a6f-c777-4079-816d-a25bd634b5be","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845201634Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:14.845210683Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845384228Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"io task","client":"0","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845420772Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"io task","client":"2","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845431731Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"io task","client":"1","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.84550456Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"task":"crutest"} {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845530266Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345} {"msg":"d6631a6f-c777-4079-816d-a25bd634b5be active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845555213Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845564651Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"client":"0","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.84557429Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"client":"1","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845583728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"client":"2","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845593377Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"io task","client":"2","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845618343Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"io task","client":"1","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845628812Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"io task","client":"0","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"ds_connection connected from Ok(127.0.0.1:45081)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845663857Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"io task","client":"2","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"ds_connection connected from Ok(127.0.0.1:64251)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845674835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"io task","client":"1","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"ds_connection connected from Ok(127.0.0.1:49428)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.845736916Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"io task","client":"0","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846306613Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"client":"1","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846469679Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"client":"0","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846485447Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"client":"2","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.84659929Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846716433Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"[0]R flush_numbers[0..12]: [89, 84, 89, 88, 94, 93, 93, 89, 88, 88, 89, 90]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846797741Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"[0]R generation[0..12]: [48, 44, 48, 47, 51, 50, 50, 48, 47, 47, 48, 50]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846808809Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846819758Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"[1]R flush_numbers[0..12]: [89, 84, 89, 88, 94, 93, 93, 89, 88, 88, 89, 90]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846830006Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"[1]R generation[0..12]: [48, 44, 48, 47, 51, 50, 50, 48, 47, 47, 48, 50]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846840155Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846850933Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"[2]R flush_numbers[0..12]: [89, 84, 89, 88, 94, 93, 93, 89, 88, 88, 89, 90]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846863281Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"[2]R generation[0..12]: [48, 44, 48, 47, 51, 50, 50, 48, 47, 47, 48, 50]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.84687353Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846884298Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846893667Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846903346Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Next flush: 95","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846912554Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846921803Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"":"downstairs","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846931411Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.84694013Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"client":"0","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846949529Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"client":"1","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:12:14.846958937Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"client":"2","":"downstairs","session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846971676Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846980774Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"d6631a6f-c777-4079-816d-a25bd634b5be is now active with session: 391bedd0-3978-4299-950f-d1d4a90eac71","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846990583Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.846999522Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.84700728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345,"session_id":"391bedd0-3978-4299-950f-d1d4a90eac71"} {"msg":"Activated sub_volume d6631a6f-c777-4079-816d-a25bd634b5be","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:14.847017489Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 93 len 32768 data: 5 6 7 7 8 7 7 7 02/30 Write block 203 len 20480 data: 2 6 5 5 4 03/30 Read block 128 len 4096 04/30 Read block 423 len 16384 05/30 Write block 437 len 28672 data: 5 6 5 6 8 10 8 06/30 Write block 9 len 24576 data: 6 7 7 7 6 5 07/30 Read block 472 len 16384 08/30 Read block 553 len 16384 09/30 Write block 196 len 8192 data: 5 6 10/30 Read block 229 len 8192 11/30 Read block 154 len 28672 12/30 Write block 388 len 32768 data: 4 5 5 6 5 5 4 4 13/30 Read block 273 len 40960 14/30 Write block 438 len 40960 data: 7 6 7 9 11 9 8 10 10 9 15/30 Read block 201 len 12288 16/30 Flush 17/30 Write block 241 len 8192 data: 7 5 18/30 Flush 19/30 Write block 132 len 40960 data: 4 6 5 4 3 3 4 7 9 10 20/30 Flush 21/30 Write block 162 len 40960 data: 7 9 8 8 6 5 4 6 5 5 22/30 Read block 179 len 16384 23/30 Write block 0 len 20480 data: 2 3 3 3 5 24/30 Read block 339 len 8192 25/30 Write block 238 len 20480 data: 8 8 8 8 6 26/30 Write block 430 len 8192 data: 3 4 27/30 Write block 95 len 36864 data: 8 8 9 8 8 8 8 6 5 28/30 Read block 189 len 20480 29/30 Write block 389 len 20480 data: 6 6 7 6 6 30/30 Read block 292 len 40960 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Write 5 Sent Done Done false 1002 Acked Read 1 Sent Done Done false 1003 Acked Read 4 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 2 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 10 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 5 Sent Done Sent false 1029 Acked Read 10 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 2 32 Done 0 30 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:14.856648881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1345} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 52 52 89 95 95 F T T <--- 1 020-039 44 44 44 84 84 84 F F F 2 040-059 48 48 48 89 89 89 F F F 3 060-079 47 47 47 88 88 88 F F F 4 080-099 51 52 52 94 95 95 F T T <--- 5 100-119 50 52 52 93 95 95 F T T <--- 6 120-139 50 52 52 93 97 97 F F F <--- 7 140-159 48 52 52 89 97 97 F F F <--- 8 160-179 47 47 47 88 88 88 F T T <--- 9 180-199 47 52 52 88 95 95 F F F <--- 10 200-219 48 52 52 89 95 95 F F F <--- 11 220-239 50 50 50 90 90 90 F T T <--- 12 240-259 50 52 52 92 96 96 F T T <--- 13 260-279 45 45 45 85 85 85 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 50 50 50 93 93 93 F F F 16 320-339 50 50 50 93 93 93 F F F 17 340-359 48 48 48 89 89 89 F F F 18 360-379 51 51 51 94 94 94 F F F 19 380-399 51 52 52 94 95 95 F T T <--- 20 400-419 48 48 48 89 89 89 F F F 21 420-439 48 52 52 89 95 95 F T T <--- 22 440-459 50 52 52 93 95 95 F F F <--- 23 460-479 51 51 51 94 94 94 F F F 24 480-499 51 51 51 94 94 94 F F F 25 500-519 51 51 51 94 94 94 F F F 26 520-539 44 44 44 81 81 81 F F F 27 540-559 51 51 51 94 94 94 F F F 28 560-579 51 51 51 94 94 94 F F F 29 580-599 43 43 43 80 80 80 F F F Max gen: 52, Max flush: 97 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 98 98 98 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 48 48 48 89 89 89 F F F 3 060-079 47 47 47 88 88 88 F F F 4 080-099 53 53 53 98 98 98 F F F 5 100-119 53 53 53 98 98 98 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 52 52 52 97 97 97 F F F 8 160-179 53 53 53 98 98 98 F F F 9 180-199 52 52 52 95 95 95 F F F 10 200-219 52 52 52 95 95 95 F F F 11 220-239 53 53 53 98 98 98 F F F 12 240-259 53 53 53 98 98 98 F F F 13 260-279 45 45 45 85 85 85 F F F 14 280-299 32 32 32 61 61 61 F F F 15 300-319 50 50 50 93 93 93 F F F 16 320-339 50 50 50 93 93 93 F F F 17 340-359 48 48 48 89 89 89 F F F 18 360-379 51 51 51 94 94 94 F F F 19 380-399 53 53 53 98 98 98 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 53 53 53 98 98 98 F F F 22 440-459 52 52 52 95 95 95 F F F 23 460-479 51 51 51 94 94 94 F F F 24 480-499 51 51 51 94 94 94 F F F 25 500-519 51 51 51 94 94 94 F F F 26 520-539 44 44 44 81 81 81 F F F 27 540-559 51 51 51 94 94 94 F F F 28 560-579 51 51 51 94 94 94 F F F 29 580-599 43 43 43 80 80 80 F F F Max gen: 53, Max flush: 98 Begin loop 27 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.113846781Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:19.121021682Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.121079444Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.127978805Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.128234268Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.128264084Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.128272522Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.128283731Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353} {"msg":"Crucible 58f929ba-d7b6-4290-9dd7-7a33e4472cb5 has session id: 2e812f49-7e1f-49d2-92db-557211611f40","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.128667345Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Upstairs opts: Upstairs UUID: 58f929ba-d7b6-4290-9dd7-7a33e4472cb5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.1286986Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Crucible stats registered with UUID: 58f929ba-d7b6-4290-9dd7-7a33e4472cb5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.128711218Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:19.128720477Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.128886523Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"io task","client":"1","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.128922038Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"io task","client":"0","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.128935046Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"io task","client":"2","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129013414Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"task":"crutest"} {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129031502Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353} {"msg":"58f929ba-d7b6-4290-9dd7-7a33e4472cb5 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129063137Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129072516Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"client":"0","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129082224Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"client":"1","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129096702Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"client":"2","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129106161Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"io task","client":"2","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129116189Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"io task","client":"1","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129126028Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"io task","client":"0","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"ds_connection connected from Ok(127.0.0.1:34542)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129144995Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"io task","client":"2","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"ds_connection connected from Ok(127.0.0.1:63156)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129186969Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"io task","client":"1","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"ds_connection connected from Ok(127.0.0.1:38912)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129241121Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"io task","client":"0","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.12986108Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"client":"1","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129885127Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"client":"0","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.129948597Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"client":"2","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130180874Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130196591Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"[0]R flush_numbers[0..12]: [98, 84, 89, 88, 98, 98, 97, 97, 98, 95, 95, 98]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130246624Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"[0]R generation[0..12]: [53, 44, 48, 47, 53, 53, 52, 52, 53, 52, 52, 53]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130257632Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130268671Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"[1]R flush_numbers[0..12]: [98, 84, 89, 88, 98, 98, 97, 97, 98, 95, 95, 98]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130279039Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"[1]R generation[0..12]: [53, 44, 48, 47, 53, 53, 52, 52, 53, 52, 52, 53]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130289358Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130300096Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"[2]R flush_numbers[0..12]: [98, 84, 89, 88, 98, 98, 97, 97, 98, 95, 95, 98]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130310355Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"[2]R generation[0..12]: [53, 44, 48, 47, 53, 53, 52, 52, 53, 52, 52, 53]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130320553Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130331462Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.13034081Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130350519Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Next flush: 99","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130359937Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130370976Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"":"downstairs","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130380614Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-04-15T18:12:19.130389343Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"client":"0","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130404131Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"client":"1","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.13041364Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"client":"2","":"downstairs","session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130423168Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130432267Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"58f929ba-d7b6-4290-9dd7-7a33e4472cb5 is now active with session: 2e812f49-7e1f-49d2-92db-557211611f40","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130442165Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130451174Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130458603Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353,"session_id":"2e812f49-7e1f-49d2-92db-557211611f40"} {"msg":"Activated sub_volume 58f929ba-d7b6-4290-9dd7-7a33e4472cb5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:19.130467792Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 298 len 28672 data: 5 4 4 4 4 5 6 03/30 Read block 539 len 4096 04/30 Write block 49 len 4096 data: 5 05/30 Read block 10 len 20480 06/30 Read block 24 len 8192 07/30 Write block 460 len 8192 data: 7 5 08/30 Write block 563 len 36864 data: 5 5 4 4 5 4 4 6 7 09/30 Write block 556 len 4096 data: 6 10/30 Flush 11/30 Write block 217 len 32768 data: 5 6 6 4 3 4 5 6 12/30 Write block 488 len 32768 data: 2 3 4 4 4 4 6 7 13/30 Flush 14/30 Flush 15/30 Write block 176 len 16384 data: 6 5 6 6 16/30 Flush 17/30 Write block 114 len 16384 data: 5 5 7 7 18/30 Write block 356 len 20480 data: 7 7 6 5 6 19/30 Read block 337 len 8192 20/30 Read block 91 len 40960 21/30 Flush 22/30 Write block 455 len 24576 data: 7 9 9 10 9 8 23/30 Write block 176 len 28672 data: 7 6 7 7 7 6 7 24/30 Read block 405 len 32768 25/30 Read block 98 len 20480 26/30 Write block 564 len 8192 data: 6 5 27/30 Write block 290 len 40960 data: 4 5 6 6 5 3 4 5 6 5 28/30 Read block 317 len 28672 29/30 Write block 575 len 12288 data: 7 8 6 30/30 Write block 74 len 12288 data: 4 4 5 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 7 Sent Done Done false 1002 Acked Read 1 Sent Done Done false 1003 Acked Write 1 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Read 2 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 2 2 33 Done 0 27 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:19.138890351Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1353} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 98 98 98 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 48 54 54 89 100 100 F F F <--- 3 060-079 47 47 47 88 88 88 F T T <--- 4 080-099 53 53 53 98 98 98 F F F 5 100-119 53 54 54 98 104 104 F F F <--- 6 120-139 52 52 52 97 97 97 F F F 7 140-159 52 52 52 97 97 97 F F F 8 160-179 53 54 54 98 103 103 F T T <--- 9 180-199 52 52 52 95 95 95 F T T <--- 10 200-219 52 54 54 95 101 101 F F F <--- 11 220-239 53 54 54 98 101 101 F F F <--- 12 240-259 53 53 53 98 98 98 F F F 13 260-279 45 45 45 85 85 85 F F F 14 280-299 32 54 54 61 100 100 F T T <--- 15 300-319 50 54 54 93 100 100 F F F <--- 16 320-339 50 50 50 93 93 93 F F F 17 340-359 48 54 54 89 104 104 F F F <--- 18 360-379 51 54 54 94 104 104 F F F <--- 19 380-399 53 53 53 98 98 98 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 53 53 53 98 98 98 F F F 22 440-459 52 52 52 95 95 95 F T T <--- 23 460-479 51 54 54 94 100 100 F T T <--- 24 480-499 51 54 54 94 101 101 F F F <--- 25 500-519 51 51 51 94 94 94 F F F 26 520-539 44 44 44 81 81 81 F F F 27 540-559 51 54 54 94 100 100 F F F <--- 28 560-579 51 54 54 94 100 100 F T T <--- 29 580-599 43 43 43 80 80 80 F F F Max gen: 54, Max flush: 104 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 98 98 98 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 54 54 54 100 100 100 F F F 3 060-079 55 55 55 105 105 105 F F F 4 080-099 53 53 53 98 98 98 F F F 5 100-119 54 54 54 104 104 104 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 52 52 52 97 97 97 F F F 8 160-179 55 55 55 105 105 105 F F F 9 180-199 55 55 55 105 105 105 F F F 10 200-219 54 54 54 101 101 101 F F F 11 220-239 54 54 54 101 101 101 F F F 12 240-259 53 53 53 98 98 98 F F F 13 260-279 45 45 45 85 85 85 F F F 14 280-299 55 55 55 105 105 105 F F F 15 300-319 54 54 54 100 100 100 F F F 16 320-339 50 50 50 93 93 93 F F F 17 340-359 54 54 54 104 104 104 F F F 18 360-379 54 54 54 104 104 104 F F F 19 380-399 53 53 53 98 98 98 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 53 53 53 98 98 98 F F F 22 440-459 55 55 55 105 105 105 F F F 23 460-479 55 55 55 105 105 105 F F F 24 480-499 54 54 54 101 101 101 F F F 25 500-519 51 51 51 94 94 94 F F F 26 520-539 44 44 44 81 81 81 F F F 27 540-559 54 54 54 100 100 100 F F F 28 560-579 55 55 55 105 105 105 F F F 29 580-599 43 43 43 80 80 80 F F F Max gen: 55, Max flush: 105 Begin loop 28 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.399210614Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.407596518Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.407904943Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.407933509Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.407942198Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.407953376Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361} {"msg":"Crucible e277edf7-8750-4bcd-9a7e-8f566c001986 has session id: 628e7d3f-d5d8-450e-9588-a466eff2b97d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408365116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Upstairs opts: Upstairs UUID: e277edf7-8750-4bcd-9a7e-8f566c001986, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408389392Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Crucible stats registered with UUID: e277edf7-8750-4bcd-9a7e-8f566c001986","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408399201Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:23.408407939Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408573895Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"io task","client":"1","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408596232Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"io task","client":"0","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408606221Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"io task","client":"2","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408738991Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"task":"crutest"} {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408757128Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361} {"msg":"e277edf7-8750-4bcd-9a7e-8f566c001986 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408795693Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408804951Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"client":"0","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.4088146Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"client":"1","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408826108Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"client":"2","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408835497Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"io task","client":"2","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408848905Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"io task","client":"1","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408893748Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"io task","client":"0","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"ds_connection connected from Ok(127.0.0.1:46263)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408930643Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"io task","client":"2","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"ds_connection connected from Ok(127.0.0.1:65458)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.40895124Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"io task","client":"1","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"ds_connection connected from Ok(127.0.0.1:40347)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.408983255Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"io task","client":"0","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.40963867Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"client":"2","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.409687702Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"client":"0","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.40970126Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"client":"1","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.409936356Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.409946155Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"[0]R flush_numbers[0..12]: [98, 84, 100, 105, 98, 104, 97, 97, 105, 105, 101, 101]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.409966102Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"[0]R generation[0..12]: [53, 44, 54, 55, 53, 54, 52, 52, 55, 55, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.40997636Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.409987339Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"[1]R flush_numbers[0..12]: [98, 84, 100, 105, 98, 104, 97, 97, 105, 105, 101, 101]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.409997897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"[1]R generation[0..12]: [53, 44, 54, 55, 53, 54, 52, 52, 55, 55, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410008146Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410021424Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"[2]R flush_numbers[0..12]: [98, 84, 100, 105, 98, 104, 97, 97, 105, 105, 101, 101]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410031682Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"[2]R generation[0..12]: [53, 44, 54, 55, 53, 54, 52, 52, 55, 55, 54, 54]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410041781Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410052459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410061638Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410071226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Next flush: 106","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410080455Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410089734Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"":"downstairs","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410099222Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410107921Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"client":"0","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.41011742Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"client":"1","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410126888Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"client":"2","":"downstairs","session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410136537Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410145486Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"Wait for a query_work_queue command to finish before sending IO 628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"e277edf7-8750-4bcd-9a7e-8f566c001986 is now active with session: 628e7d3f-d5d8-450e-9588-a466eff2b97d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410159693Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410169472Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361,"session_id":"628e7d3f-d5d8-450e-9588-a466eff2b97d"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.410178401Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361} {"msg":"Activated sub_volume e277edf7-8750-4bcd-9a7e-8f566c001986","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:23.41018656Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 255 len 4096 data: 3 03/30 Flush 04/30 Write block 551 len 32768 data: 7 6 6 7 7 7 6 5 05/30 Read block 277 len 16384 06/30 Write block 592 len 28672 data: 8 7 6 4 4 3 2 07/30 Flush 08/30 Write block 538 len 20480 data: 6 6 5 5 5 09/30 Write block 223 len 12288 data: 6 7 6 10/30 Write block 82 len 40960 data: 7 5 6 7 6 8 7 7 6 6 11/30 Write block 174 len 4096 data: 6 12/30 Write block 41 len 36864 data: 6 5 5 4 7 7 6 6 6 13/30 Read block 598 len 4096 14/30 Read block 556 len 20480 15/30 Write block 192 len 16384 data: 5 5 5 5 16/30 Write block 371 len 8192 data: 4 5 17/30 Read block 75 len 40960 18/30 Read block 285 len 36864 19/30 Write block 303 len 36864 data: 6 7 6 7 3 3 4 5 5 20/30 Read block 535 len 16384 21/30 Read block 478 len 8192 22/30 Read block 206 len 8192 23/30 Write block 201 len 28672 data: 5 3 3 7 6 6 5 24/30 Read block 121 len 32768 25/30 Write block 54 len 4096 data: 4 26/30 Read block 590 len 4096 27/30 Write block 50 len 20480 data: 4 4 4 2 5 28/30 Read block 215 len 4096 29/30 Read block 467 len 8192 30/30 Read block 105 len 24576 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Done Sent Done false 1001 Acked Write 1 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 8 Done Sent Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Write 10 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Write 4 Done Sent Done false 1015 Acked Write 2 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 1 31 Done 30 0 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1006)) None Some(JobId(1006)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:23.418860012Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1361} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 98 98 98 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 54 54 54 100 100 100 T F T <--- 3 060-079 55 55 55 105 105 105 F F F 4 080-099 53 53 53 98 98 98 T F T <--- 5 100-119 54 54 54 104 104 104 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 52 52 52 97 97 97 F F F 8 160-179 55 55 55 105 105 105 T F T <--- 9 180-199 55 55 55 105 105 105 T F T <--- 10 200-219 54 54 54 101 101 101 T F T <--- 11 220-239 54 54 54 101 101 101 T F T <--- 12 240-259 56 53 56 107 98 107 F F F <--- 13 260-279 45 45 45 85 85 85 F F F 14 280-299 55 55 55 105 105 105 F F F 15 300-319 54 54 54 100 100 100 T F T <--- 16 320-339 50 50 50 93 93 93 F F F 17 340-359 54 54 54 104 104 104 F F F 18 360-379 54 54 54 104 104 104 T F T <--- 19 380-399 53 53 53 98 98 98 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 53 53 53 98 98 98 F F F 22 440-459 55 55 55 105 105 105 F F F 23 460-479 55 55 55 105 105 105 F F F 24 480-499 54 54 54 101 101 101 F F F 25 500-519 51 51 51 94 94 94 F F F 26 520-539 44 44 44 81 81 81 T F T <--- 27 540-559 56 54 56 108 100 108 T F T <--- 28 560-579 55 55 55 105 105 105 F F F 29 580-599 56 43 56 108 80 108 F F F <--- Max gen: 56, Max flush: 108 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 98 98 98 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 57 57 57 109 109 109 F F F 3 060-079 55 55 55 105 105 105 F F F 4 080-099 57 57 57 109 109 109 F F F 5 100-119 54 54 54 104 104 104 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 52 52 52 97 97 97 F F F 8 160-179 57 57 57 109 109 109 F F F 9 180-199 57 57 57 109 109 109 F F F 10 200-219 57 57 57 109 109 109 F F F 11 220-239 57 57 57 109 109 109 F F F 12 240-259 56 56 56 107 107 107 F F F 13 260-279 45 45 45 85 85 85 F F F 14 280-299 55 55 55 105 105 105 F F F 15 300-319 57 57 57 109 109 109 F F F 16 320-339 50 50 50 93 93 93 F F F 17 340-359 54 54 54 104 104 104 F F F 18 360-379 57 57 57 109 109 109 F F F 19 380-399 53 53 53 98 98 98 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 53 53 53 98 98 98 F F F 22 440-459 55 55 55 105 105 105 F F F 23 460-479 55 55 55 105 105 105 F F F 24 480-499 54 54 54 101 101 101 F F F 25 500-519 51 51 51 94 94 94 F F F 26 520-539 57 57 57 109 109 109 F F F 27 540-559 57 57 57 109 109 109 F F F 28 560-579 55 55 55 105 105 105 F F F 29 580-599 56 56 56 108 108 108 F F F Max gen: 57, Max flush: 109 Begin loop 29 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.66819921Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.676599742Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.676922975Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.676950991Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.67695962Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.676971098Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369} {"msg":"Crucible 0c07a1a6-8b72-473f-828f-5f8d6a07a7cf has session id: 21474209-05f5-48ef-a77d-c2525117848b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677289291Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Upstairs opts: Upstairs UUID: 0c07a1a6-8b72-473f-828f-5f8d6a07a7cf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677312088Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Crucible stats registered with UUID: 0c07a1a6-8b72-473f-828f-5f8d6a07a7cf","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677322477Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:27.677331565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677497121Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"io task","client":"0","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677520308Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"io task","client":"1","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677532306Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"io task","client":"2","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677662097Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"task":"crutest"} {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677679074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369} {"msg":"0c07a1a6-8b72-473f-828f-5f8d6a07a7cf active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677715879Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677725148Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"client":"0","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677734736Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"client":"1","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677744215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"client":"2","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677753633Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"io task","client":"2","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677763352Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"io task","client":"0","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677773051Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"io task","client":"1","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"ds_connection connected from Ok(127.0.0.1:33628)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677830872Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"io task","client":"0","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"ds_connection connected from Ok(127.0.0.1:55089)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677856558Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"io task","client":"1","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"ds_connection connected from Ok(127.0.0.1:40594)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.677872416Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"io task","client":"2","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678588321Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"client":"1","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678602329Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"client":"2","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678613178Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"client":"0","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678896636Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678906105Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"[0]R flush_numbers[0..12]: [98, 84, 109, 105, 109, 104, 97, 97, 109, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678924822Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"[0]R generation[0..12]: [53, 44, 57, 55, 57, 54, 52, 52, 57, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678935171Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678946079Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"[1]R flush_numbers[0..12]: [98, 84, 109, 105, 109, 104, 97, 97, 109, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678956538Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"[1]R generation[0..12]: [53, 44, 57, 55, 57, 54, 52, 52, 57, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678966816Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678977395Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"[2]R flush_numbers[0..12]: [98, 84, 109, 105, 109, 104, 97, 97, 109, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678987703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"[2]R generation[0..12]: [53, 44, 57, 55, 57, 54, 52, 52, 57, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.678997862Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.67900858Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679018139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679027757Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Next flush: 110","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679036936Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679046185Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"":"downstairs","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679056933Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679065582Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"client":"0","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.67907512Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"client":"1","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679084719Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"client":"2","":"downstairs","session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679094488Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679103476Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"0c07a1a6-8b72-473f-828f-5f8d6a07a7cf is now active with session: 21474209-05f5-48ef-a77d-c2525117848b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679113215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679121764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369,"session_id":"21474209-05f5-48ef-a77d-c2525117848b"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679130742Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369} {"msg":"Activated sub_volume 0c07a1a6-8b72-473f-828f-5f8d6a07a7cf","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:27.679138661Z"Wait for a query_work_queue command to finish before sending IO ,"hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 379 len 20480 data: 6 5 4 4 5 02/30 Write block 97 len 16384 data: 10 9 9 9 03/30 Read block 35 len 40960 04/30 Read block 354 len 36864 05/30 Read block 70 len 16384 06/30 Write block 245 len 24576 data: 6 6 5 4 3 4 07/30 Write block 261 len 36864 data: 5 8 10 10 10 10 10 9 8 08/30 Read block 457 len 36864 09/30 Read block 213 len 40960 10/30 Flush 11/30 Write block 279 len 12288 data: 5 5 5 12/30 Read block 159 len 20480 13/30 Write block 158 len 8192 data: 5 6 14/30 Write block 163 len 24576 data: 10 9 9 7 6 5 15/30 Write block 527 len 36864 data: 5 7 8 7 6 6 7 6 7 16/30 Write block 468 len 16384 data: 6 7 7 6 17/30 Read block 460 len 4096 18/30 Flush 19/30 Flush 20/30 Write block 526 len 40960 data: 5 6 8 9 8 7 7 8 7 8 21/30 Flush 22/30 Write block 590 len 28672 data: 7 7 9 8 7 5 5 23/30 Read block 506 len 28672 24/30 Read block 177 len 40960 25/30 Read block 429 len 4096 26/30 Read block 335 len 20480 27/30 Write block 262 len 20480 data: 9 11 11 11 11 28/30 Read block 314 len 28672 29/30 Write block 497 len 16384 data: 7 6 6 5 30/30 Read block 352 len 20480 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Write 6 Done Sent Done false 1006 Acked Write 9 Done Sent Done false 1007 Acked Read 9 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Write 2 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Write 4 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 10 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Read 7 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Write 4 Done Sent Done false 1029 Acked Read 5 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 0 27 Done 29 4 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:27.688486755Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1369} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 98 98 98 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 57 57 57 109 109 109 F F F 3 060-079 55 55 55 105 105 105 F F F 4 080-099 58 57 58 110 109 110 F T F <--- 5 100-119 58 54 58 110 104 110 F T F <--- 6 120-139 52 52 52 97 97 97 F F F 7 140-159 58 52 58 111 97 111 F F F <--- 8 160-179 58 57 58 111 109 111 F F F <--- 9 180-199 57 57 57 109 109 109 F F F 10 200-219 57 57 57 109 109 109 F F F 11 220-239 57 57 57 109 109 109 F F F 12 240-259 58 56 58 110 107 110 F F F <--- 13 260-279 58 45 58 111 85 111 T F T <--- 14 280-299 58 55 58 111 105 111 F F F <--- 15 300-319 57 57 57 109 109 109 F F F 16 320-339 50 50 50 93 93 93 F F F 17 340-359 54 54 54 104 104 104 F F F 18 360-379 58 57 58 110 109 110 F T F <--- 19 380-399 58 53 58 110 98 110 F T F <--- 20 400-419 48 48 48 89 89 89 F F F 21 420-439 53 53 53 98 98 98 F F F 22 440-459 55 55 55 105 105 105 F F F 23 460-479 58 55 58 111 105 111 F F F <--- 24 480-499 54 54 54 101 101 101 T F T <--- 25 500-519 51 51 51 94 94 94 T F T <--- 26 520-539 58 57 58 113 109 113 F F F <--- 27 540-559 57 57 57 109 109 109 F F F 28 560-579 55 55 55 105 105 105 F F F 29 580-599 56 56 56 108 108 108 T F T <--- Max gen: 58, Max flush: 113 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 98 98 98 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 57 57 57 109 109 109 F F F 3 060-079 55 55 55 105 105 105 F F F 4 080-099 58 58 58 110 110 110 F F F 5 100-119 58 58 58 110 110 110 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 58 58 58 111 111 111 F F F 8 160-179 58 58 58 111 111 111 F F F 9 180-199 57 57 57 109 109 109 F F F 10 200-219 57 57 57 109 109 109 F F F 11 220-239 57 57 57 109 109 109 F F F 12 240-259 58 58 58 110 110 110 F F F 13 260-279 59 59 59 114 114 114 F F F 14 280-299 58 58 58 111 111 111 F F F 15 300-319 57 57 57 109 109 109 F F F 16 320-339 50 50 50 93 93 93 F F F 17 340-359 54 54 54 104 104 104 F F F 18 360-379 58 58 58 110 110 110 F F F 19 380-399 58 58 58 110 110 110 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 53 53 53 98 98 98 F F F 22 440-459 55 55 55 105 105 105 F F F 23 460-479 58 58 58 111 111 111 F F F 24 480-499 59 59 59 114 114 114 F F F 25 500-519 59 59 59 114 114 114 F F F 26 520-539 58 58 58 113 113 113 F F F 27 540-559 57 57 57 109 109 109 F F F 28 560-579 55 55 55 105 105 105 F F F 29 580-599 59 59 59 114 114 114 F F F Max gen: 59, Max flush: 114 Begin loop 30 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.937783459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.945923359Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.946221455Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.946249441Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.94625795Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.946269978Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377} {"msg":"Crucible 8ab180ab-4825-44f8-8ab2-085672c7cf70 has session id: 6ccc243b-8042-48db-884f-7a763fb0f236","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.946590841Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Upstairs opts: Upstairs UUID: 8ab180ab-4825-44f8-8ab2-085672c7cf70, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.946648623Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Crucible stats registered with UUID: 8ab180ab-4825-44f8-8ab2-085672c7cf70","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.946680868Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:31.946695666Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.946858542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"io task","client":"1","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.946895427Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"io task","client":"0","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.946906415Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"io task","client":"2","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947036356Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"task":"crutest"} {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947054074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377} {"msg":"8ab180ab-4825-44f8-8ab2-085672c7cf70 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947086559Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947095808Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"client":"0","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947105606Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"client":"1","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947115285Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"client":"2","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947124833Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"io task","client":"2","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947139451Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"io task","client":"0","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947151149Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"io task","client":"1","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"ds_connection connected from Ok(127.0.0.1:61553)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947220529Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"io task","client":"1","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"ds_connection connected from Ok(127.0.0.1:60821)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947231368Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"io task","client":"0","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"ds_connection connected from Ok(127.0.0.1:65342)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947256134Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"io task","client":"2","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.947988897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"client":"0","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948023912Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"client":"2","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948041749Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"client":"1","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948297092Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948331877Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"[0]R flush_numbers[0..12]: [98, 84, 109, 105, 110, 110, 97, 111, 111, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948396317Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"[0]R generation[0..12]: [53, 44, 57, 55, 58, 58, 52, 58, 58, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948407346Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948418264Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"[1]R flush_numbers[0..12]: [98, 84, 109, 105, 110, 110, 97, 111, 111, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948428783Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"[1]R generation[0..12]: [53, 44, 57, 55, 58, 58, 52, 58, 58, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948438881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.94844975Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"[2]R flush_numbers[0..12]: [98, 84, 109, 105, 110, 110, 97, 111, 111, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948462828Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"[2]R generation[0..12]: [53, 44, 57, 55, 58, 58, 52, 58, 58, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948473006Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948483775Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Max found gen is 60","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948493013Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Generation requested: 60 >= found:60","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948502542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Next flush: 115","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948511761Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948521009Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"":"downstairs","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948530678Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948539417Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"client":"0","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948549055Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"client":"1","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948558574Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"client":"2","":"downstairs","session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948568442Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","levelWait for a query_work_queue command to finish before sending IO ":30,"time":"2026-04-15T18:12:31.94858116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"8ab180ab-4825-44f8-8ab2-085672c7cf70 is now active with session: 6ccc243b-8042-48db-884f-7a763fb0f236","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948591089Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948600088Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948608766Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377,"session_id":"6ccc243b-8042-48db-884f-7a763fb0f236"} {"msg":"Activated sub_volume 8ab180ab-4825-44f8-8ab2-085672c7cf70","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:31.948617725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 567 len 40960 data: 6 5 5 7 8 7 5 6 8 9 02/30 Read block 382 len 4096 03/30 Write block 167 len 24576 data: 7 6 7 6 6 6 04/30 Read block 39 len 24576 05/30 Read block 219 len 20480 06/30 Read block 315 len 32768 07/30 Flush 08/30 Write block 53 len 20480 data: 3 6 4 5 5 09/30 Read block 3 len 40960 10/30 Write block 163 len 12288 data: 11 10 10 11/30 Read block 257 len 16384 12/30 Write block 330 len 40960 data: 6 7 7 5 6 5 7 5 6 6 13/30 Write block 42 len 16384 data: 6 6 5 8 14/30 Read block 565 len 40960 15/30 Write block 272 len 8192 data: 5 6 16/30 Read block 503 len 16384 17/30 Read block 36 len 12288 18/30 Flush 19/30 Write block 538 len 28672 data: 7 7 6 6 6 5 3 20/30 Read block 388 len 40960 21/30 Read block 243 len 32768 22/30 Write block 497 len 40960 data: 8 7 7 6 5 5 3 2 4 5 23/30 Read block 98 len 16384 24/30 Write block 166 len 16384 data: 8 8 7 8 25/30 Flush 26/30 Write block 171 len 8192 data: 7 7 27/30 Write block 326 len 40960 data: 5 7 6 5 7 8 8 6 7 6 28/30 Read block 120 len 12288 29/30 Read block 586 len 8192 30/30 Write block 251 len 24576 data: 5 4 2 2 4 3 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 1 Done Sent Done false 1002 Acked Write 6 Done Sent Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Write 4 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Write 2 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:31.958314888Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1377} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 98 98 98 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 60 57 60 116 109 116 F F F <--- 3 060-079 55 55 55 105 105 105 F F F 4 080-099 58 58 58 110 110 110 F F F 5 100-119 58 58 58 110 110 110 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 58 58 58 111 111 111 F F F 8 160-179 60 58 60 117 111 117 T F T <--- 9 180-199 57 57 57 109 109 109 F F F 10 200-219 57 57 57 109 109 109 F F F 11 220-239 57 57 57 109 109 109 F F F 12 240-259 58 58 58 110 110 110 T F T <--- 13 260-279 60 59 60 116 114 116 F F F <--- 14 280-299 58 58 58 111 111 111 F F F 15 300-319 57 57 57 109 109 109 F F F 16 320-339 60 50 60 116 93 116 T F T <--- 17 340-359 54 54 54 104 104 104 F F F 18 360-379 58 58 58 110 110 110 F F F 19 380-399 58 58 58 110 110 110 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 53 53 53 98 98 98 F F F 22 440-459 55 55 55 105 105 105 F F F 23 460-479 58 58 58 111 111 111 F F F 24 480-499 60 59 60 117 114 117 F F F <--- 25 500-519 60 59 60 117 114 117 F F F <--- 26 520-539 60 58 60 117 113 117 F F F <--- 27 540-559 60 57 60 117 109 117 F F F <--- 28 560-579 60 55 60 115 105 115 F T F <--- 29 580-599 59 59 59 114 114 114 F F F Max gen: 60, Max flush: 117 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 98 98 98 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 60 60 60 116 116 116 F F F 3 060-079 55 55 55 105 105 105 F F F 4 080-099 58 58 58 110 110 110 F F F 5 100-119 58 58 58 110 110 110 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 58 58 58 111 111 111 F F F 8 160-179 61 61 61 118 118 118 F F F 9 180-199 57 57 57 109 109 109 F F F 10 200-219 57 57 57 109 109 109 F F F 11 220-239 57 57 57 109 109 109 F F F 12 240-259 61 61 61 118 118 118 F F F 13 260-279 60 60 60 116 116 116 F F F 14 280-299 58 58 58 111 111 111 F F F 15 300-319 57 57 57 109 109 109 F F F 16 320-339 61 61 61 118 118 118 F F F 17 340-359 54 54 54 104 104 104 F F F 18 360-379 58 58 58 110 110 110 F F F 19 380-399 58 58 58 110 110 110 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 53 53 53 98 98 98 F F F 22 440-459 55 55 55 105 105 105 F F F 23 460-479 58 58 58 111 111 111 F F F 24 480-499 60 60 60 117 117 117 F F F 25 500-519 60 60 60 117 117 117 F F F 26 520-539 60 60 60 117 117 117 F F F 27 540-559 60 60 60 117 117 117 F F F 28 560-579 60 60 60 115 115 115 F F F 29 580-599 59 59 59 114 114 114 F F F Max gen: 61, Max flush: 118 Begin loop 31 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.2029775Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:36.210190955Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.210246537Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.217241505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.217497057Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.217526083Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.217534672Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.21754622Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385} {"msg":"Crucible 7384bfe8-710c-4217-aaa1-66e7cc4bb3ec has session id: a0d86967-14b2-4968-b1c9-c3e2159b560f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.217870583Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Upstairs opts: Upstairs UUID: 7384bfe8-710c-4217-aaa1-66e7cc4bb3ec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.217899658Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Crucible stats registered with UUID: 7384bfe8-710c-4217-aaa1-66e7cc4bb3ec","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.217910887Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:36.217920145Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218079482Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"io task","client":"0","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218117857Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"io task","client":"2","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218128835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"io task","client":"1","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218254857Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"task":"crutest"} {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218271654Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385} {"msg":"7384bfe8-710c-4217-aaa1-66e7cc4bb3ec active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218305139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218314618Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"client":"0","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218324166Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"client":"1","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218333635Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"client":"2","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218343014Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"io task","client":"2","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218352752Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"io task","client":"0","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218362471Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"io task","client":"1","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"ds_connection connected from Ok(127.0.0.1:64452)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218459797Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"io task","client":"2","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"ds_connection connected from Ok(127.0.0.1:40540)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218470845Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"io task","client":"1","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"ds_connection connected from Ok(127.0.0.1:49091)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.218552903Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"io task","client":"0","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219135558Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"client":"2","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219154135Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"client":"1","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219245972Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"client":"0","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219427665Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219444663Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"[0]R flush_numbers[0..12]: [98, 84, 116, 105, 110, 110, 97, 111, 118, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219498215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"[0]R generation[0..12]: [53, 44, 60, 55, 58, 58, 52, 58, 61, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219509293Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219520232Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"[1]R flush_numbers[0..12]: [98, 84, 116, 105, 110, 110, 97, 111, 118, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.21953066Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"[1]R generation[0..12]: [53, 44, 60, 55, 58, 58, 52, 58, 61, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219540989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219551797Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"[2]R flush_numbers[0..12]: [98, 84, 116, 105, 110, 110, 97, 111, 118, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219562225Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"[2]R generation[0..12]: [53, 44, 60, 55, 58, 58, 52, 58, 61, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219572504Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219583232Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219592641Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.2196022Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Next flush: 119","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219612868Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219622247Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"":"downstairs","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219631955Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219640634Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"client":"0","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219654952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"client":"1","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219664381Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"client":"2","":"downstairs","session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219674119Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219683158Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"7384bfe8-710c-4217-aaa1-66e7cc4bb3ec is now active with session: a0d86967-14b2-4968-b1c9-c3e2159b560f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219692956Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219701915Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219709434Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385,"session_id":"a0d86967-14b2-4968-b1c9-c3e2159b560f"} {"msg":"Activated sub_volume 7384bfe8-710c-4217-aaa1-66e7cc4bb3ec","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:36.219718523Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 2 len 20480 data: 4 4 6 6 6 02/30 Write block 163 len 20480 data: 12 11 11 9 9 03/30 Read block 371 len 20480 04/30 Read block 472 len 12288 05/30 Write block 365 len 28672 data: 3 3 3 3 3 4 5 06/30 Read block 353 len 16384 07/30 Read block 403 len 4096 08/30 Write block 569 len 16384 data: 6 8 9 8 09/30 Write block 97 len 8192 data: 11 10 10/30 Write block 467 len 8192 data: 7 7 11/30 Read block 52 len 24576 12/30 Read block 508 len 4096 13/30 Read block 332 len 32768 14/30 Read block 130 len 36864 15/30 Write block 103 len 20480 data: 6 4 6 7 8 16/30 Write block 103 len 4096 data: 7 17/30 Write block 282 len 36864 data: 4 6 6 5 5 4 5 4 5 18/30 Flush 19/30 Write block 472 len 40960 data: 5 5 6 4 3 3 3 4 4 4 20/30 Write block 473 len 16384 data: 6 7 5 4 21/30 Write block 311 len 40960 data: 6 5 7 7 7 10 9 9 8 9 22/30 Write block 63 len 36864 data: 4 4 4 3 2 2 2 2 3 23/30 Read block 60 len 24576 24/30 Read block 437 len 20480 25/30 Read block 9 len 32768 26/30 Write block 427 len 20480 data: 5 4 4 4 5 27/30 Write block 352 len 4096 data: 10 28/30 Write block 293 len 4096 data: 7 29/30 Read block 412 len 8192 30/30 Read block 319 len 32768 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Read 3 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Write 4 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Read 9 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Write 10 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Write 5 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Read 8 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 0 27 Done 3 30 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:36.228134922Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1385} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 62 62 98 119 119 T F F <--- 1 020-039 44 44 44 84 84 84 F F F 2 040-059 60 60 60 116 116 116 F F F 3 060-079 55 55 55 105 105 105 F T T <--- 4 080-099 58 62 62 110 119 119 F F F <--- 5 100-119 58 62 62 110 119 119 F F F <--- 6 120-139 52 52 52 97 97 97 F F F 7 140-159 58 58 58 111 111 111 F F F 8 160-179 61 62 62 118 119 119 T F F <--- 9 180-199 57 57 57 109 109 109 F F F 10 200-219 57 57 57 109 109 109 F F F 11 220-239 57 57 57 109 109 109 F F F 12 240-259 61 61 61 118 118 118 F F F 13 260-279 60 60 60 116 116 116 F F F 14 280-299 58 62 62 111 119 119 F T T <--- 15 300-319 57 57 57 109 109 109 F T T <--- 16 320-339 61 61 61 118 118 118 F T T <--- 17 340-359 54 54 54 104 104 104 F T T <--- 18 360-379 58 62 62 110 119 119 F F F <--- 19 380-399 58 58 58 110 110 110 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 53 53 53 98 98 98 F T T <--- 22 440-459 55 55 55 105 105 105 F F F 23 460-479 58 62 62 111 119 119 F T T <--- 24 480-499 60 60 60 117 117 117 F T T <--- 25 500-519 60 60 60 117 117 117 F F F 26 520-539 60 60 60 117 117 117 F F F 27 540-559 60 60 60 117 117 117 F F F 28 560-579 60 62 62 115 119 119 F F F <--- 29 580-599 59 59 59 114 114 114 F F F Max gen: 62, Max flush: 119 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 119 119 119 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 60 60 60 116 116 116 F F F 3 060-079 63 63 63 120 120 120 F F F 4 080-099 62 62 62 119 119 119 F F F 5 100-119 62 62 62 119 119 119 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 58 58 58 111 111 111 F F F 8 160-179 62 62 62 119 119 119 F F F 9 180-199 57 57 57 109 109 109 F F F 10 200-219 57 57 57 109 109 109 F F F 11 220-239 57 57 57 109 109 109 F F F 12 240-259 61 61 61 118 118 118 F F F 13 260-279 60 60 60 116 116 116 F F F 14 280-299 63 63 63 120 120 120 F F F 15 300-319 63 63 63 120 120 120 F F F 16 320-339 63 63 63 120 120 120 F F F 17 340-359 63 63 63 120 120 120 F F F 18 360-379 62 62 62 119 119 119 F F F 19 380-399 58 58 58 110 110 110 F F F 20 400-419 48 48 48 89 89 89 F F F 21 420-439 63 63 63 120 120 120 F F F 22 440-459 55 55 55 105 105 105 F F F 23 460-479 63 63 63 120 120 120 F F F 24 480-499 63 63 63 120 120 120 F F F 25 500-519 60 60 60 117 117 117 F F F 26 520-539 60 60 60 117 117 117 F F F 27 540-559 60 60 60 117 117 117 F F F 28 560-579 62 62 62 119 119 119 F F F 29 580-599 59 59 59 114 114 114 F F F Max gen: 63, Max flush: 120 Begin loop 32 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.487712494Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.496033998Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.496311197Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.496339503Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.496350792Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.4963622Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393} {"msg":"Crucible 3ac93786-daff-4076-8960-b85d080fb1e6 has session id: 69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.496660166Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Upstairs opts: Upstairs UUID: 3ac93786-daff-4076-8960-b85d080fb1e6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.496715648Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Crucible stats registered with UUID: 3ac93786-daff-4076-8960-b85d080fb1e6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.496727507Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:40.496736785Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.496899891Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"io task","client":"0","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.496928177Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"io task","client":"1","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.496938546Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"io task","client":"2","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.49704748Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"task":"crutest"} {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497070866Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393} {"msg":"3ac93786-daff-4076-8960-b85d080fb1e6 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497089954Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497098992Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"client":"0","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497109291Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"client":"1","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497118899Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"client":"2","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497133597Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"io task","client":"2","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497146235Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"io task","client":"0","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497156124Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"io task","client":"1","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"ds_connection connected from Ok(127.0.0.1:56732)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497240762Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"io task","client":"1","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"ds_connection connected from Ok(127.0.0.1:49309)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.49732187Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"io task","client":"2","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"ds_connection connected from Ok(127.0.0.1:36043)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497341667Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"io task","client":"0","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497871649Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"client":"2","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.497893326Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"client":"1","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498024727Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"client":"0","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498180394Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498190663Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"[0]R flush_numbers[0..12]: [119, 84, 116, 120, 119, 119, 97, 111, 119, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498319284Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"[0]R generation[0..12]: [62, 44, 60, 63, 62, 62, 52, 58, 62, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498330672Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498341631Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"[1]R flush_numbers[0..12]: [119, 84, 116, 120, 119, 119, 97, 111, 119, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498354509Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"[1]R generation[0..12]: [62, 44, 60, 63, 62, 62, 52, 58, 62, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498364867Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498375756Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"[2]R flush_numbers[0..12]: [119, 84, 116, 120, 119, 119, 97, 111, 119, 109, 109, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498386394Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"[2]R generation[0..12]: [62, 44, 60, 63, 62, 62, 52, 58, 62, 57, 57, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498396753Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498407481Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.49841681Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498426698Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Next flush: 121","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498435917Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498445226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"":"downstairs","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498454974Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498463753Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"client":"0","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498473461Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"client":"1","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.49848338Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"client":"2","":"downstairs","session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498493119Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498503397Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"3ac93786-daff-4076-8960-b85d080fb1e6 is now active with session: 69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498513256Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498526544Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498534223Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393,"session_id":"69674fb4-c8bd-4ea2-81c6-01a9fb40dc7d"} {"msg":"Activated sub_volume 3ac93786-daff-4076-8960-b85d080fb1e6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:40.498543281Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 373 len 4096 data: 6 03/30 Read block 319 len 36864 04/30 Read block 545 len 8192 05/30 Read block 424 len 24576 06/30 Read block 56 len 20480 07/30 Read block 267 len 32768 08/30 Flush 09/30 Write block 256 len 16384 data: 4 3 3 4 10/30 Write block 376 len 28672 data: 7 7 6 7 6 5 5 11/30 Flush 12/30 Write block 288 len 24576 data: 6 5 6 6 7 8 13/30 Read block 83 len 8192 14/30 Read block 155 len 4096 15/30 Write block 438 len 32768 data: 8 7 8 10 12 10 9 11 16/30 Write block 94 len 16384 data: 7 9 9 12 17/30 Read block 119 len 40960 18/30 Write block 198 len 32768 data: 7 7 6 6 4 4 8 7 19/30 Write block 398 len 12288 data: 5 4 4 20/30 Read block 468 len 8192 21/30 Read block 539 len 32768 22/30 Read block 545 len 36864 23/30 Read block 481 len 40960 24/30 Write block 58 len 20480 data: 3 3 3 4 3 25/30 Read block 415 len 32768 26/30 Read block 576 len 4096 27/30 Read block 159 len 20480 28/30 Read block 548 len 8192 29/30 Read block 356 len 32768 30/30 Read block 454 len 8192 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 1 Done Sent Done false 1002 Acked Read 9 Done Sent Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Read 6 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Write 7 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Write 4 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Read 10 Done Sent Done false 1023 Acked Write 5 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Read 2 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1000)) Some(JobId(1010)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:40.507428223Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1393} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 119 119 119 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 60 60 60 116 116 116 T F T <--- 3 060-079 63 63 63 120 120 120 T F T <--- 4 080-099 62 62 62 119 119 119 T F T <--- 5 100-119 62 62 62 119 119 119 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 58 58 58 111 111 111 F F F 8 160-179 62 62 62 119 119 119 F F F 9 180-199 57 57 57 109 109 109 T F T <--- 10 200-219 57 57 57 109 109 109 T F T <--- 11 220-239 57 57 57 109 109 109 F F F 12 240-259 64 61 64 123 118 123 F F F <--- 13 260-279 60 60 60 116 116 116 F F F 14 280-299 63 63 63 120 120 120 T F T <--- 15 300-319 63 63 63 120 120 120 F F F 16 320-339 63 63 63 120 120 120 F F F 17 340-359 63 63 63 120 120 120 F F F 18 360-379 64 62 64 123 119 123 F F F <--- 19 380-399 64 58 64 123 110 123 T F T <--- 20 400-419 48 48 48 89 89 89 T F T <--- 21 420-439 63 63 63 120 120 120 T F T <--- 22 440-459 55 55 55 105 105 105 T F T <--- 23 460-479 63 63 63 120 120 120 F F F 24 480-499 63 63 63 120 120 120 F F F 25 500-519 60 60 60 117 117 117 F F F 26 520-539 60 60 60 117 117 117 F F F 27 540-559 60 60 60 117 117 117 F F F 28 560-579 62 62 62 119 119 119 F F F 29 580-599 59 59 59 114 114 114 F F F Max gen: 64, Max flush: 123 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 119 119 119 F F F 1 020-039 44 44 44 84 84 84 F F F 2 040-059 65 65 65 124 124 124 F F F 3 060-079 65 65 65 124 124 124 F F F 4 080-099 65 65 65 124 124 124 F F F 5 100-119 62 62 62 119 119 119 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 58 58 58 111 111 111 F F F 8 160-179 62 62 62 119 119 119 F F F 9 180-199 65 65 65 124 124 124 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 57 57 57 109 109 109 F F F 12 240-259 64 64 64 123 123 123 F F F 13 260-279 60 60 60 116 116 116 F F F 14 280-299 65 65 65 124 124 124 F F F 15 300-319 63 63 63 120 120 120 F F F 16 320-339 63 63 63 120 120 120 F F F 17 340-359 63 63 63 120 120 120 F F F 18 360-379 64 64 64 123 123 123 F F F 19 380-399 65 65 65 124 124 124 F F F 20 400-419 65 65 65 124 124 124 F F F 21 420-439 65 65 65 124 124 124 F F F 22 440-459 65 65 65 124 124 124 F F F 23 460-479 63 63 63 120 120 120 F F F 24 480-499 63 63 63 120 120 120 F F F 25 500-519 60 60 60 117 117 117 F F F 26 520-539 60 60 60 117 117 117 F F F 27 540-559 60 60 60 117 117 117 F F F 28 560-579 62 62 62 119 119 119 F F F 29 580-599 59 59 59 114 114 114 F F F Max gen: 65, Max flush: 124 Begin loop 33 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.75868965Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.766815142Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767098101Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767124077Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767132536Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767144044Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401} {"msg":"Crucible d84f777c-26a9-4c9c-8068-60a959a6045b has session id: d8b8b793-6875-4cd3-b221-fb0c37a406b4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767490423Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Upstairs opts: Upstairs UUID: d84f777c-26a9-4c9c-8068-60a959a6045b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767507991Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Crucible stats registered with UUID: d84f777c-26a9-4c9c-8068-60a959a6045b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767517839Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:44.767526798Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767681056Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"io task","client":"0","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.76771615Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"io task","client":"1","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767727249Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"io task","client":"2","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767821315Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"task":"crutest"} {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767846151Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401} {"msg":"d84f777c-26a9-4c9c-8068-60a959a6045b active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767864449Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767873318Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"client":"0","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767882976Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"client":"1","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767892625Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"client":"2","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767902123Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"io task","client":"2","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767911882Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"io task","client":"0","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767931199Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"io task","client":"1","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:61770)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.767942367Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"io task","client":"2","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:57006)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.768087566Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"io task","client":"0","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:49132)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.768112543Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"io task","client":"1","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.768566526Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"client":"2","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.768660273Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"client":"0","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.768673631Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"client":"1","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.768886359Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.768942671Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"[0]R flush_numbers[0..12]: [119, 84, 124, 124, 124, 119, 97, 111, 119, 124, 124, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769009831Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"[0]R generation[0..12]: [62, 44, 65, 65, 65, 62, 52, 58, 62, 65, 65, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.76902268Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769034138Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"[1]R flush_numbers[0..12]: [119, 84, 124, 124, 124, 119, 97, 111, 119, 124, 124, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769044806Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"[1]R generation[0..12]: [62, 44, 65, 65, 65, 62, 52, 58, 62, 65, 65, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769055255Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769066233Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"[2]R flush_numbers[0..12]: [119, 84, 124, 124, 124, 119, 97, 111, 119, 124, 124, 109]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769076802Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"[2]R generation[0..12]: [62, 44, 65, 65, 65, 62, 52, 58, 62, 65, 65, 57]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.76908699Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769097929Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Max found gen is 66","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769107287Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769119445Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Next flush: 125","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769128764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769138193Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"":"downstairs","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769148001Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.76915664Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"client":"0","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769175567Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"client":"1","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769185046Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"client":"2","":"downstairs","session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769194764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769203753Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"d84f777c-26a9-4c9c-8068-60a959a6045b is now active with session: d8b8b793-6875-4cd3-b221-fb0c37a406b4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769213492Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.7692225Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769233969Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401,"session_id":"d8b8b793-6875-4cd3-b221-fb0c37a406b4"} {"msg":"Activated sub_volume d84f777c-26a9-4c9c-8068-60a959a6045b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:44.769243137Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 225 len 28672 data: 7 9 9 10 9 8 4 02/30 Write block 515 len 20480 data: 5 5 6 5 5 03/30 Read block 51 len 24576 04/30 Read block 268 len 40960 05/30 Write block 177 len 40960 data: 7 8 8 8 7 8 7 6 5 5 06/30 Read block 347 len 8192 07/30 Write block 496 len 24576 data: 6 9 8 8 7 6 08/30 Write block 309 len 12288 data: 5 6 7 09/30 Write block 525 len 12288 data: 4 6 7 10/30 Write block 101 len 4096 data: 9 11/30 Write block 437 len 40960 data: 6 9 8 9 11 13 11 10 12 11 12/30 Write block 532 len 20480 data: 8 9 8 9 7 13/30 Read block 70 len 28672 14/30 Read block 94 len 32768 15/30 Write block 508 len 12288 data: 5 5 4 16/30 Write block 23 len 12288 data: 2 4 5 17/30 Read block 237 len 24576 18/30 Write block 509 len 40960 data: 6 5 4 4 5 7 6 6 7 6 19/30 Write block 177 len 40960 data: 8 9 9 9 8 9 8 7 6 6 20/30 Flush 21/30 Write block 376 len 8192 data: 8 8 22/30 Read block 496 len 36864 23/30 Write block 55 len 20480 data: 5 6 6 4 4 24/30 Write block 284 len 36864 data: 7 6 6 5 7 6 7 7 8 25/30 Read block 358 len 40960 26/30 Flush 27/30 Write block 195 len 12288 data: 6 6 7 28/30 Write block 190 len 32768 data: 4 5 6 6 6 7 7 8 29/30 Read block 409 len 40960 30/30 Read block 451 len 4096 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Sent Done false 1001 Acked Write 5 Done Sent Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Read 10 Done Sent Done false 1004 Acked Write 10 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Write 3 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Write 3 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Write 10 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Read 10 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 8 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Read 1 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:44.777947365Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1401} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 119 119 119 F F F 1 020-039 66 44 66 125 84 125 F F F <--- 2 040-059 66 65 66 126 124 126 F F F <--- 3 060-079 65 65 65 124 124 124 F F F 4 080-099 65 65 65 124 124 124 F F F 5 100-119 66 62 66 125 119 125 F F F <--- 6 120-139 52 52 52 97 97 97 F F F 7 140-159 58 58 58 111 111 111 F F F 8 160-179 66 62 66 125 119 125 F F F <--- 9 180-199 66 65 66 125 124 125 T F T <--- 10 200-219 65 65 65 124 124 124 F F F 11 220-239 66 57 66 125 109 125 F F F <--- 12 240-259 64 64 64 123 123 123 F F F 13 260-279 60 60 60 116 116 116 F F F 14 280-299 66 65 66 126 124 126 F F F <--- 15 300-319 66 63 66 125 120 125 F F F <--- 16 320-339 63 63 63 120 120 120 F F F 17 340-359 63 63 63 120 120 120 F F F 18 360-379 66 64 66 126 123 126 F F F <--- 19 380-399 65 65 65 124 124 124 F F F 20 400-419 65 65 65 124 124 124 F F F 21 420-439 66 65 66 125 124 125 F F F <--- 22 440-459 66 65 66 125 124 125 F F F <--- 23 460-479 63 63 63 120 120 120 F F F 24 480-499 66 63 66 125 120 125 F F F <--- 25 500-519 66 60 66 125 117 125 F F F <--- 26 520-539 66 60 66 125 117 125 F F F <--- 27 540-559 60 60 60 117 117 117 F F F 28 560-579 62 62 62 119 119 119 F F F 29 580-599 59 59 59 114 114 114 F F F Max gen: 66, Max flush: 126 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 119 119 119 F F F 1 020-039 66 66 66 125 125 125 F F F 2 040-059 66 66 66 126 126 126 F F F 3 060-079 65 65 65 124 124 124 F F F 4 080-099 65 65 65 124 124 124 F F F 5 100-119 66 66 66 125 125 125 F F F 6 120-139 52 52 52 97 97 97 F F F 7 140-159 58 58 58 111 111 111 F F F 8 160-179 66 66 66 125 125 125 F F F 9 180-199 67 67 67 127 127 127 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 66 66 66 125 125 125 F F F 12 240-259 64 64 64 123 123 123 F F F 13 260-279 60 60 60 116 116 116 F F F 14 280-299 66 66 66 126 126 126 F F F 15 300-319 66 66 66 125 125 125 F F F 16 320-339 63 63 63 120 120 120 F F F 17 340-359 63 63 63 120 120 120 F F F 18 360-379 66 66 66 126 126 126 F F F 19 380-399 65 65 65 124 124 124 F F F 20 400-419 65 65 65 124 124 124 F F F 21 420-439 66 66 66 125 125 125 F F F 22 440-459 66 66 66 125 125 125 F F F 23 460-479 63 63 63 120 120 120 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 66 66 66 125 125 125 F F F 26 520-539 66 66 66 125 125 125 F F F 27 540-559 60 60 60 117 117 117 F F F 28 560-579 62 62 62 119 119 119 F F F 29 580-599 59 59 59 114 114 114 F F F Max gen: 67, Max flush: 127 Begin loop 34 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.027669496Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.035865948Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036162664Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036198419Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036209088Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036220596Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409} {"msg":"Crucible 3448dffc-aecc-4f18-8158-933d16b4f3da has session id: 5e601873-19c9-4724-9e04-a7b9640ec6a0","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036495416Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Upstairs opts: Upstairs UUID: 3448dffc-aecc-4f18-8158-933d16b4f3da, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036571885Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Crucible stats registered with UUID: 3448dffc-aecc-4f18-8158-933d16b4f3da","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036596521Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:49.03660661Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036735611Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"io task","client":"2","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036770176Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"io task","client":"0","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036780324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"io task","client":"1","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036890308Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"task":"crutest"} {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036908565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409} {"msg":"3448dffc-aecc-4f18-8158-933d16b4f3da active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036931932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036941151Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"client":"0","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036950889Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"client":"1","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036962848Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"client":"2","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036972576Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"io task","client":"2","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036982475Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"io task","client":"0","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.036992263Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"io task","client":"1","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"ds_connection connected from Ok(127.0.0.1:53899)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.037179736Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"io task","client":"2","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"ds_connection connected from Ok(127.0.0.1:40691)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.037205352Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"io task","client":"0","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"ds_connection connected from Ok(127.0.0.1:65134)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.03721661Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"io task","client":"1","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.037791916Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"client":"1","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.037816253Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"client":"0","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.037884433Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"client":"2","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038093602Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.03810979Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"[0]R flush_numbers[0..12]: [119, 125, 126, 124, 124, 125, 97, 111, 125, 127, 124, 125]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038163942Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"[0]R generation[0..12]: [62, 66, 66, 65, 65, 66, 52, 58, 66, 67, 65, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.0381751Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038190348Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"[1]R flush_numbers[0..12]: [119, 125, 126, 124, 124, 125, 97, 111, 125, 127, 124, 125]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038201057Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"[1]R generation[0..12]: [62, 66, 66, 65, 65, 66, 52, 58, 66, 67, 65, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038211335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038222053Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"[2]R flush_numbers[0..12]: [119, 125, 126, 124, 124, 125, 97, 111, 125, 127, 124, 125]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038232492Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"[2]R generation[0..12]: [62, 66, 66, 65, 65, 66, 52, 58, 66, 67, 65, 66]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.03824276Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038253469Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038262798Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038272336Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Next flush: 128","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038281605Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038290743Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"":"downstairs","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038300302Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038308871Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"client":"0","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038318379Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"client":"1","":"downstairs","session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038327908Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"client":"2","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038343746Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038352864Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"3448dffc-aecc-4f18-8158-933d16b4f3da is now active with session: 5e601873-19c9-4724-9e04-a7b9640ec6a0","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038362743Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038371652Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.03837937Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409,"session_id":"5e601873-19c9-4724-9e04-a7b9640ec6a0"} {"msg":"Activated sub_volume 3448dffc-aecc-4f18-8158-933d16b4f3da","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:49.038388579Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 383 len 12288 data: 6 5 4 02/30 Write block 317 len 16384 data: 10 10 9 10 03/30 Write block 433 len 16384 data: 5 7 7 6 04/30 Read block 457 len 12288 05/30 Read block 237 len 12288 06/30 Read block 246 len 20480 07/30 Read block 30 len 8192 08/30 Read block 151 len 20480 09/30 Write block 229 len 28672 data: 10 9 5 4 3 3 3 10/30 Read block 371 len 24576 11/30 Write block 248 len 28672 data: 5 4 5 6 5 3 3 12/30 Write block 472 len 20480 data: 6 7 8 6 5 13/30 Read block 325 len 32768 14/30 Read block 252 len 24576 15/30 Write block 386 len 12288 data: 4 4 5 16/30 Flush 17/30 Flush 18/30 Write block 49 len 20480 data: 7 5 5 5 4 19/30 Flush 20/30 Write block 564 len 40960 data: 7 6 5 7 6 7 9 10 9 6 21/30 Write block 56 len 12288 data: 7 7 5 22/30 Write block 64 len 40960 data: 5 5 4 3 3 3 3 4 4 4 23/30 Read block 219 len 24576 24/30 Write block 64 len 24576 data: 6 6 5 4 4 4 25/30 Write block 527 len 32768 data: 8 9 10 9 8 9 10 9 26/30 Write block 18 len 12288 data: 4 4 3 27/30 Write block 126 len 32768 data: 3 3 2 3 4 3 5 7 28/30 Read block 499 len 12288 29/30 Write block 315 len 20480 data: 8 11 11 11 10 30/30 Write block 189 len 24576 data: 3 5 6 7 7 7 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Sent false 1001 Acked Write 4 Done Done Sent false 1002 Acked Write 4 Done Done Sent false 1003 Acked Read 3 Done Done Sent false 1004 Acked Read 3 Done Done Sent false 1005 Acked Read 5 Done Done Sent false 1006 Acked Read 2 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Write 7 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Read 6 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Write 3 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Write 6 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 30 34 Done 28 28 0 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:49.046705873Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1409} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 62 62 62 119 119 119 T T F <--- 1 020-039 66 66 66 125 125 125 T T F <--- 2 040-059 68 68 66 130 130 126 T T F <--- 3 060-079 65 65 65 124 124 124 T T F <--- 4 080-099 65 65 65 124 124 124 F F F 5 100-119 66 66 66 125 125 125 F F F 6 120-139 52 52 52 97 97 97 T T F <--- 7 140-159 58 58 58 111 111 111 F F F 8 160-179 66 66 66 125 125 125 F F F 9 180-199 67 67 67 127 127 127 T T F <--- 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 66 128 128 125 F F F <--- 12 240-259 68 68 64 128 128 123 F F F <--- 13 260-279 60 60 60 116 116 116 F F F 14 280-299 66 66 66 126 126 126 F F F 15 300-319 68 68 66 128 128 125 T T F <--- 16 320-339 68 68 63 128 128 120 F F F <--- 17 340-359 63 63 63 120 120 120 F F F 18 360-379 66 66 66 126 126 126 F F F 19 380-399 68 68 65 128 128 124 F F F <--- 20 400-419 65 65 65 124 124 124 F F F 21 420-439 68 68 66 128 128 125 F F F <--- 22 440-459 66 66 66 125 125 125 F F F 23 460-479 68 68 63 128 128 120 F F F <--- 24 480-499 66 66 66 125 125 125 F F F 25 500-519 66 66 66 125 125 125 F F F 26 520-539 66 66 66 125 125 125 T T F <--- 27 540-559 60 60 60 117 117 117 F F F 28 560-579 62 62 62 119 119 119 T T F <--- 29 580-599 59 59 59 114 114 114 F F F Max gen: 68, Max flush: 130 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 131 131 131 F F F 1 020-039 69 69 69 131 131 131 F F F 2 040-059 69 69 69 131 131 131 F F F 3 060-079 69 69 69 131 131 131 F F F 4 080-099 65 65 65 124 124 124 F F F 5 100-119 66 66 66 125 125 125 F F F 6 120-139 69 69 69 131 131 131 F F F 7 140-159 58 58 58 111 111 111 F F F 8 160-179 66 66 66 125 125 125 F F F 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 68 68 68 128 128 128 F F F 13 260-279 60 60 60 116 116 116 F F F 14 280-299 66 66 66 126 126 126 F F F 15 300-319 69 69 69 131 131 131 F F F 16 320-339 68 68 68 128 128 128 F F F 17 340-359 63 63 63 120 120 120 F F F 18 360-379 66 66 66 126 126 126 F F F 19 380-399 68 68 68 128 128 128 F F F 20 400-419 65 65 65 124 124 124 F F F 21 420-439 68 68 68 128 128 128 F F F 22 440-459 66 66 66 125 125 125 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 66 66 66 125 125 125 F F F 26 520-539 69 69 69 131 131 131 F F F 27 540-559 60 60 60 117 117 117 F F F 28 560-579 69 69 69 131 131 131 F F F 29 580-599 59 59 59 114 114 114 F F F Max gen: 69, Max flush: 131 Begin loop 35 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.299467371Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.307630628Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.30796144Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308007463Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308017222Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.30802889Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417} {"msg":"Crucible 8191e7e5-fa36-4060-8a96-8660c28336df has session id: 56726581-8d93-4b53-a8d6-4d442467c013","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308360912Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Upstairs opts: Upstairs UUID: 8191e7e5-fa36-4060-8a96-8660c28336df, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308413744Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Crucible stats registered with UUID: 8191e7e5-fa36-4060-8a96-8660c28336df","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308426612Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:53.308435811Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308580279Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"io task","client":"0","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308602156Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"io task","client":"1","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308612045Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"io task","client":"2","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308753414Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"task":"crutest"} {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.30877929Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417} {"msg":"8191e7e5-fa36-4060-8a96-8660c28336df active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308798118Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308806956Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"client":"0","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308816525Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"client":"1","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308825994Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"client":"2","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308835562Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"io task","client":"2","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308845391Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"io task","client":"0","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308855129Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"io task","client":"1","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"ds_connection connected from Ok(127.0.0.1:39118)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308931128Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"io task","client":"2","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"ds_connection connected from Ok(127.0.0.1:62524)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308956504Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"io task","client":"1","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"ds_connection connected from Ok(127.0.0.1:33006)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.308969703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"io task","client":"0","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.309566785Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"client":"1","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.309586932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"client":"0","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.309652063Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"client":"2","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.309872571Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.309909975Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"[0]R flush_numbers[0..12]: [131, 131, 131, 131, 124, 125, 131, 111, 125, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.309948639Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"[0]R generation[0..12]: [69, 69, 69, 69, 65, 66, 69, 58, 66, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.309959558Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.309970486Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"[1]R flush_numbers[0..12]: [131, 131, 131, 131, 124, 125, 131, 111, 125, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.309980915Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"[1]R generation[0..12]: [69, 69, 69, 69, 65, 66, 69, 58, 66, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.309991063Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310001842Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"[2]R flush_numbers[0..12]: [131, 131, 131, 131, 124, 125, 131, 111, 125, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.31001245Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"[2]R generation[0..12]: [69, 69, 69, 69, 65, 66, 69, 58, 66, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310022909Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310035457Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310044665Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310054204Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Next flush: 132","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310063563Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310072771Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"":"downstairs","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.31008238Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310091069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"client":"0","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310100417Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"client":"1","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310110006Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"client":"2","":"downstairs","session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310119724Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"Wait for a query_work_queue command to finish before sending IO 56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310133352Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"8191e7e5-fa36-4060-8a96-8660c28336df is now active with session: 56726581-8d93-4b53-a8d6-4d442467c013","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310143271Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.31015227Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310159749Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417,"session_id":"56726581-8d93-4b53-a8d6-4d442467c013"} {"msg":"Activated sub_volume 8191e7e5-fa36-4060-8a96-8660c28336df","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:53.310168847Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 156 len 24576 data: 3 5 6 7 6 6 02/30 Write block 276 len 24576 data: 4 4 2 6 6 6 03/30 Write block 583 len 36864 data: 4 7 9 6 8 9 7 8 8 04/30 Read block 173 len 24576 05/30 Write block 42 len 24576 data: 7 7 6 9 8 7 06/30 Read block 257 len 20480 07/30 Write block 267 len 4096 data: 11 08/30 Write block 453 len 28672 data: 5 7 8 10 10 11 10 09/30 Read block 323 len 8192 10/30 Write block 433 len 28672 data: 6 8 8 7 7 10 9 11/30 Write block 84 len 32768 data: 7 8 7 9 8 8 7 7 12/30 Write block 569 len 4096 data: 8 13/30 Flush 14/30 Write block 338 len 40960 data: 7 7 7 8 9 8 7 9 9 7 15/30 Write block 391 len 40960 data: 8 7 7 5 5 4 4 6 5 5 16/30 Write block 279 len 28672 data: 7 7 7 5 7 8 7 17/30 Write block 99 len 24576 data: 10 10 10 7 8 5 18/30 Flush 19/30 Write block 327 len 36864 data: 8 7 6 8 9 9 7 8 7 20/30 Read block 563 len 40960 21/30 Read block 282 len 40960 22/30 Read block 246 len 28672 23/30 Write block 93 len 36864 data: 6 8 10 10 13 11 11 11 11 24/30 Write block 557 len 32768 data: 7 6 5 6 6 6 6 8 25/30 Write block 360 len 4096 data: 7 26/30 Read block 275 len 28672 27/30 Write block 122 len 8192 data: 8 6 28/30 Read block 396 len 32768 29/30 Read block 393 len 20480 30/30 Read block 115 len 20480 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Write 9 Done Done Sent false 1003 Acked Read 6 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Read 5 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Write 7 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Write 7 Done Done Sent false 1010 Acked Write 8 Done Done Sent false 1011 Acked Write 1 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Write 7 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Read 10 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 28 29 Done 30 29 2 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) Some(JobId(1017)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:53.319237342Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1417} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 131 131 131 F F F 1 020-039 69 69 69 131 131 131 F F F 2 040-059 70 70 69 132 132 131 F F F <--- 3 060-079 69 69 69 131 131 131 F F F 4 080-099 70 70 65 133 133 124 T T F <--- 5 100-119 70 70 66 133 133 125 T T F <--- 6 120-139 69 69 69 131 131 131 T T F <--- 7 140-159 70 70 58 132 132 111 F F T <--- 8 160-179 70 70 66 132 132 125 F F T <--- 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 68 68 68 128 128 128 F F F 13 260-279 70 70 60 133 133 116 F F T <--- 14 280-299 70 70 66 133 133 126 F F T <--- 15 300-319 69 69 69 131 131 131 F F F 16 320-339 70 70 68 133 133 128 T T F <--- 17 340-359 70 70 63 133 133 120 F F F <--- 18 360-379 66 66 66 126 126 126 T T F <--- 19 380-399 70 70 68 133 133 128 F F F <--- 20 400-419 70 70 65 133 133 124 F F F <--- 21 420-439 70 70 68 132 132 128 F F F <--- 22 440-459 70 70 66 132 132 125 F F F <--- 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 66 66 66 125 125 125 F F F 26 520-539 69 69 69 131 131 131 F F F 27 540-559 60 60 60 117 117 117 T T F <--- 28 560-579 70 70 69 132 132 131 T T F <--- 29 580-599 70 70 59 132 132 114 F F F <--- Max gen: 70, Max flush: 133 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 131 131 131 F F F 1 020-039 69 69 69 131 131 131 F F F 2 040-059 70 70 70 132 132 132 F F F 3 060-079 69 69 69 131 131 131 F F F 4 080-099 71 71 71 134 134 134 F F F 5 100-119 71 71 71 134 134 134 F F F 6 120-139 71 71 71 134 134 134 F F F 7 140-159 70 70 70 132 132 132 F F F 8 160-179 70 70 70 132 132 132 F F F 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 68 68 68 128 128 128 F F F 13 260-279 70 70 70 133 133 133 F F F 14 280-299 70 70 70 133 133 133 F F F 15 300-319 69 69 69 131 131 131 F F F 16 320-339 71 71 71 134 134 134 F F F 17 340-359 70 70 70 133 133 133 F F F 18 360-379 71 71 71 134 134 134 F F F 19 380-399 70 70 70 133 133 133 F F F 20 400-419 70 70 70 133 133 133 F F F 21 420-439 70 70 70 132 132 132 F F F 22 440-459 70 70 70 132 132 132 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 66 66 66 125 125 125 F F F 26 520-539 69 69 69 131 131 131 F F F 27 540-559 71 71 71 134 134 134 F F F 28 560-579 71 71 71 134 134 134 F F F 29 580-599 70 70 70 132 132 132 F F F Max gen: 71, Max flush: 134 Begin loop 36 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.579346846Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.587566084Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.587877979Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.587904505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.587912764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.587924152Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425} {"msg":"Crucible 3a38e5a4-624c-4ed0-a780-5aa4046521ae has session id: fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588265252Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Upstairs opts: Upstairs UUID: 3a38e5a4-624c-4ed0-a780-5aa4046521ae, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588318025Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Crucible stats registered with UUID: 3a38e5a4-624c-4ed0-a780-5aa4046521ae","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588328693Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:57.588337722Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588508307Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"io task","client":"2","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588544991Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"io task","client":"1","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.58855568Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"io task","client":"0","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588633648Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"task":"crutest"} {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588651786Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425} {"msg":"3a38e5a4-624c-4ed0-a780-5aa4046521ae active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588685521Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588695049Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"client":"0","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588704678Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"client":"1","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588714367Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"client":"2","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588723865Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"io task","client":"2","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588738483Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"io task","client":"1","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588784876Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"io task","client":"0","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"ds_connection connected from Ok(127.0.0.1:49799)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588810173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"io task","client":"2","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"ds_connection connected from Ok(127.0.0.1:39066)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588889911Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"io task","client":"1","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"ds_connection connected from Ok(127.0.0.1:62394)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.588945393Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"io task","client":"0","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589401646Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"client":"2","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589473216Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"client":"0","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.58950907Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"client":"1","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589728208Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589753745Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"[0]R flush_numbers[0..12]: [131, 131, 132, 131, 134, 134, 134, 132, 132, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589811676Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"[0]R generation[0..12]: [69, 69, 70, 69, 71, 71, 71, 70, 70, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589823295Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589834413Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"[1]R flush_numbers[0..12]: [131, 131, 132, 131, 134, 134, 134, 132, 132, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589845021Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"[1]R generation[0..12]: [69, 69, 70, 69, 71, 71, 71, 70, 70, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.5898553Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589869968Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"[2]R flush_numbers[0..12]: [131, 131, 132, 131, 134, 134, 134, 132, 132, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589880646Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"[2]R generation[0..12]: [69, 69, 70, 69, 71, 71, 71, 70, 70, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589890835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589901633Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589910952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.58992033Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Next flush: 135","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589929719Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589938928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"":"downstairs","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589948426Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589957035Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"client":"0","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589968183Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"client":"1","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589977642Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"client":"2","":"downstairs","session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589987311Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.589996289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pidWait for a query_work_queue command to finish before sending IO ":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"3a38e5a4-624c-4ed0-a780-5aa4046521ae is now active with session: fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.590011357Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.590020406Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.590028155Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425,"session_id":"fcf7eeb4-8de1-4ba3-9eff-f09c6df3c9ef"} {"msg":"Activated sub_volume 3a38e5a4-624c-4ed0-a780-5aa4046521ae","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:12:57.590037253Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 542 len 20480 data: 7 6 4 4 5 02/30 Read block 459 len 16384 03/30 Write block 243 len 16384 data: 6 5 7 7 04/30 Read block 373 len 40960 05/30 Read block 31 len 24576 06/30 Read block 497 len 4096 07/30 Write block 273 len 8192 data: 7 6 08/30 Read block 307 len 40960 09/30 Write block 21 len 16384 data: 3 2 3 5 10/30 Read block 156 len 12288 11/30 Read block 352 len 20480 12/30 Write block 369 len 36864 data: 4 5 6 6 7 8 6 9 9 13/30 Read block 582 len 12288 14/30 Read block 371 len 12288 15/30 Write block 96 len 20480 data: 11 14 12 12 12 16/30 Read block 398 len 28672 17/30 Write block 13 len 20480 data: 7 6 5 5 4 18/30 Read block 224 len 24576 19/30 Write block 111 len 32768 data: 6 4 4 6 6 8 8 7 20/30 Write block 431 len 20480 data: 6 4 7 9 9 21/30 Write block 508 len 20480 data: 6 7 6 5 5 22/30 Read block 436 len 40960 23/30 Read block 535 len 8192 24/30 Write block 551 len 32768 data: 8 7 7 8 8 8 8 7 25/30 Write block 25 len 36864 data: 6 5 5 8 7 6 4 4 3 26/30 Write block 444 len 4096 data: 11 27/30 Write block 12 len 24576 data: 8 8 7 6 6 5 28/30 Write block 382 len 8192 data: 6 7 29/30 Read block 31 len 40960 30/30 Read block 556 len 12288 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Read 10 Done Done Done false 1004 Acked Read 6 Done Done Done false 1005 Acked Read 1 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Read 3 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 0 25 Done 30 5 30 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:12:57.599078492Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1425} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 131 131 131 T F T <--- 1 020-039 69 69 69 131 131 131 T F T <--- 2 040-059 70 70 70 132 132 132 F F F 3 060-079 69 69 69 131 131 131 F F F 4 080-099 71 71 71 134 134 134 T F T <--- 5 100-119 71 71 71 134 134 134 T F T <--- 6 120-139 71 71 71 134 134 134 F F F 7 140-159 70 70 70 132 132 132 F F F 8 160-179 70 70 70 132 132 132 F F F 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 68 68 68 128 128 128 T T T 13 260-279 70 70 70 133 133 133 T F T <--- 14 280-299 70 70 70 133 133 133 F F F 15 300-319 69 69 69 131 131 131 F F F 16 320-339 71 71 71 134 134 134 F F F 17 340-359 70 70 70 133 133 133 F F F 18 360-379 71 71 71 134 134 134 T F T <--- 19 380-399 70 70 70 133 133 133 T F T <--- 20 400-419 70 70 70 133 133 133 F F F 21 420-439 70 70 70 132 132 132 T F T <--- 22 440-459 70 70 70 132 132 132 T F T <--- 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 66 66 66 125 125 125 T F T <--- 26 520-539 69 69 69 131 131 131 F F F 27 540-559 71 71 71 134 134 134 T T T 28 560-579 71 71 71 134 134 134 F F F 29 580-599 70 70 70 132 132 132 F F F Max gen: 71, Max flush: 134 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 72 72 72 135 135 135 F F F 2 040-059 70 70 70 132 132 132 F F F 3 060-079 69 69 69 131 131 131 F F F 4 080-099 72 72 72 135 135 135 F F F 5 100-119 72 72 72 135 135 135 F F F 6 120-139 71 71 71 134 134 134 F F F 7 140-159 70 70 70 132 132 132 F F F 8 160-179 70 70 70 132 132 132 F F F 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 72 72 72 135 135 135 F F F 13 260-279 72 72 72 135 135 135 F F F 14 280-299 70 70 70 133 133 133 F F F 15 300-319 69 69 69 131 131 131 F F F 16 320-339 71 71 71 134 134 134 F F F 17 340-359 70 70 70 133 133 133 F F F 18 360-379 72 72 72 135 135 135 F F F 19 380-399 72 72 72 135 135 135 F F F 20 400-419 70 70 70 133 133 133 F F F 21 420-439 72 72 72 135 135 135 F F F 22 440-459 72 72 72 135 135 135 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 72 72 72 135 135 135 F F F 26 520-539 69 69 69 131 131 131 F F F 27 540-559 72 72 72 135 135 135 F F F 28 560-579 71 71 71 134 134 134 F F F 29 580-599 70 70 70 132 132 132 F F F Max gen: 72, Max flush: 135 Begin loop 37 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.854930268Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:01.862057116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.862118867Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.869113955Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.869383785Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.869410172Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.86941857Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.869432588Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433} {"msg":"Crucible ec778e77-0a94-401b-bf64-1b4f77a90d93 has session id: 4a557958-ad5c-4708-b1fb-3140973ee959","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.869776778Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Upstairs opts: Upstairs UUID: ec778e77-0a94-401b-bf64-1b4f77a90d93, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.869796795Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Crucible stats registered with UUID: ec778e77-0a94-401b-bf64-1b4f77a90d93","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.869807094Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:01.869815922Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.869981698Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"io task","client":"2","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870004245Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"io task","client":"0","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870014203Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"io task","client":"1","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870127617Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"task":"crutest"} {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870145764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433} {"msg":"ec778e77-0a94-401b-bf64-1b4f77a90d93 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870182409Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870191647Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"client":"0","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870201246Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"client":"1","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870210675Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"client":"2","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870220013Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"io task","client":"2","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870231252Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"io task","client":"1","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870277905Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"io task","client":"0","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"ds_connection connected from Ok(127.0.0.1:34250)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870288763Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"io task","client":"2","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"ds_connection connected from Ok(127.0.0.1:59120)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870298882Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"io task","client":"1","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"ds_connection connected from Ok(127.0.0.1:56950)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870414755Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"io task","client":"0","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870895165Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"client":"1","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.870935289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"client":"2","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871023026Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"client":"0","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871233015Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871242924Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"[0]R flush_numbers[0..12]: [135, 135, 132, 131, 135, 135, 134, 132, 132, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871340229Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"[0]R generation[0..12]: [72, 72, 70, 69, 72, 72, 71, 70, 70, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871351258Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871362196Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"[1]R flush_numbers[0..12]: [135, 135, 132, 131, 135, 135, 134, 132, 132, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871372655Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"[1]R generation[0..12]: [72, 72, 70, 69, 72, 72, 71, 70, 70, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871382863Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871395961Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"[2]R flush_numbers[0..12]: [135, 135, 132, 131, 135, 135, 134, 132, 132, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.87140645Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"[2]R generation[0..12]: [72, 72, 70, 69, 72, 72, 71, 70, 70, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871416918Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871427857Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Max found gen is 73","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871437105Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Generation requested: 74 >= found:73","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871446674Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Next flush: 136","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871455943Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871465141Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"":"downstairs","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.87147481Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871483499Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"client":"0","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871493017Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"client":"1","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} Wait for a query_work_queue command to finish before sending IO {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871507085Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"client":"2","":"downstairs","session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871516994Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871526012Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"ec778e77-0a94-401b-bf64-1b4f77a90d93 is now active with session: 4a557958-ad5c-4708-b1fb-3140973ee959","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871536011Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871546289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871554218Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433,"session_id":"4a557958-ad5c-4708-b1fb-3140973ee959"} {"msg":"Activated sub_volume ec778e77-0a94-401b-bf64-1b4f77a90d93","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:01.871563437Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 449 len 12288 data: 10 7 5 02/30 Write block 241 len 28672 data: 9 7 7 6 8 8 6 03/30 Write block 309 len 16384 data: 6 7 8 6 04/30 Write block 424 len 4096 data: 6 05/30 Read block 59 len 20480 06/30 Read block 420 len 16384 07/30 Write block 52 len 12288 data: 6 5 7 08/30 Read block 404 len 32768 09/30 Flush 10/30 Write block 425 len 8192 data: 5 5 11/30 Write block 140 len 32768 data: 10 11 9 10 10 11 8 7 12/30 Read block 248 len 28672 13/30 Read block 24 len 8192 14/30 Flush 15/30 Write block 75 len 16384 data: 5 6 5 6 16/30 Read block 278 len 32768 17/30 Read block 55 len 28672 18/30 Write block 535 len 20480 data: 10 8 8 8 8 19/30 Write block 271 len 8192 data: 6 6 20/30 Read block 83 len 20480 21/30 Flush 22/30 Write block 572 len 40960 data: 10 7 7 9 10 7 6 5 4 4 23/30 Read block 116 len 32768 24/30 Write block 524 len 20480 data: 5 5 7 9 10 25/30 Write block 285 len 12288 data: 8 7 6 26/30 Read block 182 len 32768 27/30 Read block 249 len 16384 28/30 Write block 78 len 24576 data: 7 5 5 6 8 6 29/30 Write block 259 len 12288 data: 5 3 6 30/30 Write block 30 len 24576 data: 7 5 5 4 4 4 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Read 5 Done Done Done false 1005 Acked Read 4 Done Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 6 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 3 3 30 Done 6 27 27 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:01.879848536Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1433} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 72 72 72 135 135 135 F T T <--- 2 040-059 70 74 74 132 136 136 F F F <--- 3 060-079 69 74 74 131 138 138 F T T <--- 4 080-099 72 72 72 135 135 135 F T T <--- 5 100-119 72 72 72 135 135 135 F F F 6 120-139 71 71 71 134 134 134 F F F 7 140-159 70 74 74 132 137 137 F F F <--- 8 160-179 70 70 70 132 132 132 F F F 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 72 74 74 135 136 136 T T T <--- 13 260-279 72 74 74 135 138 138 F T T <--- 14 280-299 70 70 70 133 133 133 F T T <--- 15 300-319 69 74 74 131 136 136 T F F <--- 16 320-339 71 71 71 134 134 134 F F F 17 340-359 70 70 70 133 133 133 F F F 18 360-379 72 72 72 135 135 135 F F F 19 380-399 72 72 72 135 135 135 F F F 20 400-419 70 70 70 133 133 133 F F F 21 420-439 72 74 74 135 137 137 T F F <--- 22 440-459 72 74 74 135 136 136 T F F <--- 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 72 72 72 135 135 135 F F F 26 520-539 69 74 74 131 138 138 F T T <--- 27 540-559 72 72 72 135 135 135 F F F 28 560-579 71 71 71 134 134 134 F T T <--- 29 580-599 70 70 70 132 132 132 F T T <--- Max gen: 74, Max flush: 138 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 74 74 74 136 136 136 F F F 3 060-079 75 75 75 139 139 139 F F F 4 080-099 75 75 75 139 139 139 F F F 5 100-119 72 72 72 135 135 135 F F F 6 120-139 71 71 71 134 134 134 F F F 7 140-159 74 74 74 137 137 137 F F F 8 160-179 70 70 70 132 132 132 F F F 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 75 75 75 139 139 139 F F F 13 260-279 75 75 75 139 139 139 F F F 14 280-299 75 75 75 139 139 139 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 71 71 71 134 134 134 F F F 17 340-359 70 70 70 133 133 133 F F F 18 360-379 72 72 72 135 135 135 F F F 19 380-399 72 72 72 135 135 135 F F F 20 400-419 70 70 70 133 133 133 F F F 21 420-439 74 74 74 137 137 137 F F F 22 440-459 74 74 74 136 136 136 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 72 72 72 135 135 135 F F F 26 520-539 75 75 75 139 139 139 F F F 27 540-559 72 72 72 135 135 135 F F F 28 560-579 75 75 75 139 139 139 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 75, Max flush: 139 Begin loop 38 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.142267712Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.150521416Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.150802215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.15083077Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.150839909Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.150851327Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441} {"msg":"Crucible 76454c11-3076-4a09-b8d7-192274db2507 has session id: 975d9dae-eeb6-42fc-aa45-e2ff57397cd0","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151158293Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Upstairs opts: Upstairs UUID: 76454c11-3076-4a09-b8d7-192274db2507, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151187378Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Crucible stats registered with UUID: 76454c11-3076-4a09-b8d7-192274db2507","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151198267Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:06.151207505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151392658Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"io task","client":"2","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151417365Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"io task","client":"1","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151427663Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"io task","client":"0","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151540257Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"task":"crutest"} {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151555684Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441} {"msg":"76454c11-3076-4a09-b8d7-192274db2507 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151573542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151582261Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"client":"0","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151591779Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"client":"1","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151601318Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"client":"2","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151610726Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"io task","client":"2","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151643942Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"io task","client":"1","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"ds_connection connected from Ok(127.0.0.1:63785)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151661459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"io task","client":"2","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151671298Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"io task","client":"0","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"ds_connection connected from Ok(127.0.0.1:57204)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151685475Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"io task","client":"1","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"ds_connection connected from Ok(127.0.0.1:41833)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.151766814Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"io task","client":"0","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152307605Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"client":"1","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.15234017Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"client":"0","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152354148Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"client":"2","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.15261169Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152641636Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"[0]R flush_numbers[0..12]: [135, 139, 136, 139, 139, 135, 134, 137, 132, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152709116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"[0]R generation[0..12]: [72, 75, 74, 75, 75, 72, 71, 74, 70, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152720074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152731123Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"[1]R flush_numbers[0..12]: [135, 139, 136, 139, 139, 135, 134, 137, 132, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152741641Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"[1]R generation[0..12]: [72, 75, 74, 75, 75, 72, 71, 74, 70, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.15275179Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152762658Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"[2]R flush_numbers[0..12]: [135, 139, 136, 139, 139, 135, 134, 137, 132, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152773077Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"[2]R generation[0..12]: [72, 75, 74, 75, 75, 72, 71, 74, 70, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152783265Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152793933Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152803182Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152812661Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Next flush: 140","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152823489Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152832768Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"":"downstairs","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152842446Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152851105Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"client":"0","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152860674Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"client":"1","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152870042Z"Wait for a query_work_queue command to finish before sending IO ,"hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"client":"2","":"downstairs","session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.15288432Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152893279Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"76454c11-3076-4a09-b8d7-192274db2507 is now active with session: 975d9dae-eeb6-42fc-aa45-e2ff57397cd0","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152903188Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152912256Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152919765Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441,"session_id":"975d9dae-eeb6-42fc-aa45-e2ff57397cd0"} {"msg":"Activated sub_volume 76454c11-3076-4a09-b8d7-192274db2507","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:06.152928894Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 163 len 40960 data: 13 12 12 10 10 8 9 7 8 8 03/30 Read block 398 len 20480 04/30 Read block 215 len 20480 05/30 Flush 06/30 Write block 352 len 36864 data: 11 9 9 8 8 8 7 6 8 07/30 Read block 322 len 40960 08/30 Read block 204 len 32768 09/30 Flush 10/30 Write block 535 len 20480 data: 11 9 9 9 9 11/30 Write block 447 len 4096 data: 10 12/30 Read block 410 len 40960 13/30 Flush 14/30 Write block 390 len 36864 data: 7 9 8 8 6 6 5 5 7 15/30 Read block 195 len 8192 16/30 Read block 376 len 28672 17/30 Read block 509 len 36864 18/30 Flush 19/30 Write block 41 len 4096 data: 7 20/30 Write block 288 len 32768 data: 8 7 8 8 9 9 6 4 21/30 Write block 89 len 24576 data: 9 8 8 6 7 9 22/30 Read block 374 len 8192 23/30 Read block 385 len 36864 24/30 Flush 25/30 Write block 531 len 32768 data: 9 10 11 10 12 10 10 10 26/30 Write block 254 len 8192 data: 4 5 27/30 Read block 536 len 24576 28/30 Read block 457 len 12288 29/30 Read block 578 len 16384 30/30 Read block 125 len 36864 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 10 Done Done Sent false 1002 Acked Read 5 Done Done Sent false 1003 Acked Read 5 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Read 10 Done Done Sent false 1007 Acked Read 8 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 5 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Read 9 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 1 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 29 28 0 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:06.162969116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1441} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 76 76 74 145 145 136 F F F <--- 3 060-079 75 75 75 139 139 139 F F F 4 080-099 76 76 75 145 145 139 F F F <--- 5 100-119 72 72 72 135 135 135 F F F 6 120-139 71 71 71 134 134 134 F F F 7 140-159 74 74 74 137 137 137 F F F 8 160-179 76 76 70 141 141 132 F F F <--- 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 75 75 75 139 139 139 T T F <--- 13 260-279 75 75 75 139 139 139 F F F 14 280-299 76 76 75 145 145 139 F F F <--- 15 300-319 74 74 74 136 136 136 F F F 16 320-339 71 71 71 134 134 134 F F F 17 340-359 76 76 70 142 142 133 F F F <--- 18 360-379 76 76 72 142 142 135 F F F <--- 19 380-399 76 76 72 144 144 135 F F F <--- 20 400-419 70 70 70 133 133 133 F F F 21 420-439 74 74 74 137 137 137 F F F 22 440-459 76 76 74 143 143 136 F F F <--- 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 72 72 72 135 135 135 F F F 26 520-539 76 76 75 143 143 139 T T F <--- 27 540-559 72 72 72 135 135 135 F F F 28 560-579 75 75 75 139 139 139 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 76, Max flush: 145 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 76 76 76 145 145 145 F F F 3 060-079 75 75 75 139 139 139 F F F 4 080-099 76 76 76 145 145 145 F F F 5 100-119 72 72 72 135 135 135 F F F 6 120-139 71 71 71 134 134 134 F F F 7 140-159 74 74 74 137 137 137 F F F 8 160-179 76 76 76 141 141 141 F F F 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 77 77 77 146 146 146 F F F 13 260-279 75 75 75 139 139 139 F F F 14 280-299 76 76 76 145 145 145 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 71 71 71 134 134 134 F F F 17 340-359 76 76 76 142 142 142 F F F 18 360-379 76 76 76 142 142 142 F F F 19 380-399 76 76 76 144 144 144 F F F 20 400-419 70 70 70 133 133 133 F F F 21 420-439 74 74 74 137 137 137 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 72 72 72 135 135 135 F F F 26 520-539 77 77 77 146 146 146 F F F 27 540-559 72 72 72 135 135 135 F F F 28 560-579 75 75 75 139 139 139 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 77, Max flush: 146 Begin loop 39 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.406280835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.414411737Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.414703284Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.414727471Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.4147358Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.414751467Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449} {"msg":"Crucible fe1d233e-9384-49b4-b503-53f96a764cff has session id: a9ce9ed3-f713-4913-8cd1-6061ebed39bb","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.41507673Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Upstairs opts: Upstairs UUID: fe1d233e-9384-49b4-b503-53f96a764cff, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415104576Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Crucible stats registered with UUID: fe1d233e-9384-49b4-b503-53f96a764cff","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415115634Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:10.415124863Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415287449Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"io task","client":"0","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415322874Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"io task","client":"2","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415334092Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"io task","client":"1","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415439027Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"task":"crutest"} {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415463193Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449} {"msg":"fe1d233e-9384-49b4-b503-53f96a764cff active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415537472Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415546861Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"client":"0","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.4155564Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"client":"1","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415566248Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"client":"2","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415575867Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"io task","client":"2","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415587855Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"io task","client":"1","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415597844Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"io task","client":"0","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:61411)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415633688Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"io task","client":"2","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:65341)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415644517Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"io task","client":"1","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:35537)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.415684851Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"io task","client":"0","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416202325Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"client":"0","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416224382Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"client":"2","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416341305Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"client":"1","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416477775Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416494703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"[0]R flush_numbers[0..12]: [135, 139, 145, 139, 145, 135, 134, 137, 141, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416607306Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"[0]R generation[0..12]: [72, 75, 76, 75, 76, 72, 71, 74, 76, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416618354Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416629313Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"[1]R flush_numbers[0..12]: [135, 139, 145, 139, 145, 135, 134, 137, 141, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416639901Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"[1]R generation[0..12]: [72, 75, 76, 75, 76, 72, 71, 74, 76, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.41665204Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416662928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"[2]R flush_numbers[0..12]: [135, 139, 145, 139, 145, 135, 134, 137, 141, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416673226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"[2]R generation[0..12]: [72, 75, 76, 75, 76, 72, 71, 74, 76, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416683535Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416694393Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Max found gen is 78","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416703632Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416713211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Next flush: 147","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416722389Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416731628Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"":"downstairs","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416741466Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416750295Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"client":"0","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416760114Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"client":"1","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416769892Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"client":"2","":"downstairs","session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416779771Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.41678879Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"fe1d233e-9384-49b4-b503-53f96a764cff is now active with session: a9ce9ed3-f713-4913-8cd1-6061ebed39bb","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416798528Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2026-04-15T18:13:10.416812786Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416820485Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449,"session_id":"a9ce9ed3-f713-4913-8cd1-6061ebed39bb"} {"msg":"Activated sub_volume fe1d233e-9384-49b4-b503-53f96a764cff","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:10.416829494Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 126 len 28672 data: 4 4 3 4 5 4 6 02/30 Read block 32 len 28672 03/30 Read block 371 len 20480 04/30 Write block 98 len 32768 data: 13 13 13 12 8 9 6 7 05/30 Read block 100 len 24576 06/30 Read block 4 len 20480 07/30 Read block 255 len 12288 08/30 Write block 532 len 12288 data: 11 12 11 09/30 Write block 507 len 32768 data: 5 7 8 7 6 6 6 8 10/30 Read block 463 len 36864 11/30 Read block 328 len 24576 12/30 Flush 13/30 Write block 250 len 24576 data: 6 7 6 4 5 6 14/30 Write block 430 len 40960 data: 5 7 5 8 10 10 8 8 11 10 15/30 Read block 42 len 36864 16/30 Read block 485 len 40960 17/30 Write block 391 len 28672 data: 10 9 9 7 7 6 6 18/30 Write block 519 len 36864 data: 6 5 6 6 5 6 6 8 10 19/30 Read block 455 len 36864 20/30 Write block 508 len 8192 data: 8 9 21/30 Read block 299 len 24576 22/30 Read block 587 len 12288 23/30 Write block 102 len 36864 data: 9 10 7 8 8 9 7 7 6 24/30 Flush 25/30 Write block 341 len 28672 data: 9 10 9 8 10 10 8 26/30 Read block 556 len 40960 27/30 Write block 557 len 40960 data: 9 8 6 7 7 7 7 9 7 6 28/30 Write block 42 len 28672 data: 8 8 7 10 9 8 7 29/30 Write block 540 len 40960 data: 7 7 8 7 5 5 6 5 7 7 30/30 Read block 241 len 24576 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Read 6 Done Done Done false 1005 Acked Read 5 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Write 3 Done Done Done false 1008 Acked Write 8 Done Done Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Read 6 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Read 9 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Read 3 Done Sent Done false 1022 Acked Write 9 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Write 10 Done Sent Sent false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 21 2 23 Done 30 9 28 67 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:10.427036002Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1449} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 76 76 76 145 145 145 T F T <--- 3 060-079 75 75 75 139 139 139 F F F 4 080-099 78 76 78 147 145 147 F T F <--- 5 100-119 78 72 78 148 135 148 F T F <--- 6 120-139 78 71 78 147 134 147 F T F <--- 7 140-159 74 74 74 137 137 137 F F F 8 160-179 76 76 76 141 141 141 F F F 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 78 77 78 148 146 148 F F F <--- 13 260-279 75 75 75 139 139 139 F F F 14 280-299 76 76 76 145 145 145 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 71 71 71 134 134 134 F F F 17 340-359 76 76 76 142 142 142 T F T <--- 18 360-379 76 76 76 142 142 142 F F F 19 380-399 78 76 78 148 144 148 F F F <--- 20 400-419 70 70 70 133 133 133 F F F 21 420-439 78 74 78 148 137 148 F F F <--- 22 440-459 76 76 76 143 143 143 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 78 72 78 148 135 148 F T F <--- 26 520-539 78 77 78 148 146 148 F T F <--- 27 540-559 72 72 72 135 135 135 T F T <--- 28 560-579 75 75 75 139 139 139 T F T <--- 29 580-599 75 75 75 139 139 139 F F F Max gen: 78, Max flush: 148 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 79 79 79 149 149 149 F F F 3 060-079 75 75 75 139 139 139 F F F 4 080-099 78 78 78 147 147 147 F F F 5 100-119 78 78 78 148 148 148 F F F 6 120-139 78 78 78 147 147 147 F F F 7 140-159 74 74 74 137 137 137 F F F 8 160-179 76 76 76 141 141 141 F F F 9 180-199 69 69 69 131 131 131 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 68 68 128 128 128 F F F 12 240-259 78 78 78 148 148 148 F F F 13 260-279 75 75 75 139 139 139 F F F 14 280-299 76 76 76 145 145 145 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 71 71 71 134 134 134 F F F 17 340-359 79 79 79 149 149 149 F F F 18 360-379 76 76 76 142 142 142 F F F 19 380-399 78 78 78 148 148 148 F F F 20 400-419 70 70 70 133 133 133 F F F 21 420-439 78 78 78 148 148 148 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 78 78 78 148 148 148 F F F 26 520-539 78 78 78 148 148 148 F F F 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 79, Max flush: 149 Begin loop 40 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.673926578Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:14.681128055Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.681189936Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.688263892Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.688532313Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.688558669Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.688567058Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.688578446Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457} {"msg":"Crucible e494ddda-472d-4520-834a-e2b378306b06 has session id: e2366f67-1b3c-4df1-82a6-fb814d76896c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.688932785Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Upstairs opts: Upstairs UUID: e494ddda-472d-4520-834a-e2b378306b06, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.688956011Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Crucible stats registered with UUID: e494ddda-472d-4520-834a-e2b378306b06","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.68896749Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:14.688976628Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689134265Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"io task","client":"2","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689157072Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"io task","client":"0","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.68916711Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"io task","client":"1","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689270195Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"task":"crutest"} {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689288793Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457} {"msg":"e494ddda-472d-4520-834a-e2b378306b06 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689331716Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689340985Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"client":"0","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689350714Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"client":"1","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689360272Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"client":"2","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689369701Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"io task","client":"2","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689389598Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"io task","client":"1","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"ds_connection connected from Ok(127.0.0.1:40090)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689403976Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"io task","client":"2","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689416404Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"io task","client":"0","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"ds_connection connected from Ok(127.0.0.1:41765)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689477725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"io task","client":"1","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"ds_connection connected from Ok(127.0.0.1:61210)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.689503851Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"io task","client":"0","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690047182Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"client":"2","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690076357Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"client":"1","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690211158Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"client":"0","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690343838Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690362176Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"[0]R flush_numbers[0..12]: [135, 139, 149, 139, 147, 148, 147, 137, 141, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690487237Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"[0]R generation[0..12]: [72, 75, 79, 75, 78, 78, 78, 74, 76, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690498196Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690509174Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"[1]R flush_numbers[0..12]: [135, 139, 149, 139, 147, 148, 147, 137, 141, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690519713Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"[1]R generation[0..12]: [72, 75, 79, 75, 78, 78, 78, 74, 76, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690530021Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.69054074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"[2]R flush_numbers[0..12]: [135, 139, 149, 139, 147, 148, 147, 137, 141, 131, 124, 128]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690551528Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"[2]R generation[0..12]: [72, 75, 79, 75, 78, 78, 78, 74, 76, 69, 65, 68]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690561856Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690575455Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690584973Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690594442Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Next flush: 150","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.69060358Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690612899Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"":"downstairs","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690622398Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690631056Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"client":"0","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690640625Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"client":"1","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690650004Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"client":"2","":"downstairs","session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} Wait for a query_work_queue command to finish before sending IO {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690663312Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.69067239Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"e494ddda-472d-4520-834a-e2b378306b06 is now active with session: e2366f67-1b3c-4df1-82a6-fb814d76896c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690682149Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690691128Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690698607Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457,"session_id":"e2366f67-1b3c-4df1-82a6-fb814d76896c"} {"msg":"Activated sub_volume e494ddda-472d-4520-834a-e2b378306b06","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:14.690707675Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 223 len 8192 data: 7 8 02/30 Read block 483 len 20480 03/30 Write block 346 len 8192 data: 11 9 04/30 Read block 321 len 28672 05/30 Write block 461 len 4096 data: 6 06/30 Read block 586 len 36864 07/30 Write block 176 len 20480 data: 8 9 10 10 10 08/30 Read block 103 len 4096 09/30 Read block 8 len 8192 10/30 Read block 25 len 24576 11/30 Read block 131 len 12288 12/30 Flush 13/30 Write block 387 len 8192 data: 5 6 14/30 Read block 14 len 12288 15/30 Read block 432 len 8192 16/30 Flush 17/30 Write block 355 len 16384 data: 9 9 9 8 18/30 Read block 156 len 16384 19/30 Read block 250 len 16384 20/30 Read block 214 len 36864 21/30 Read block 113 len 32768 22/30 Read block 322 len 16384 23/30 Read block 351 len 28672 24/30 Read block 590 len 20480 25/30 Write block 363 len 36864 data: 3 3 4 4 4 4 5 6 7 26/30 Write block 242 len 36864 data: 8 8 7 9 9 7 6 5 7 27/30 Read block 112 len 8192 28/30 Write block 403 len 4096 data: 4 29/30 Read block 132 len 20480 30/30 Write block 131 len 8192 data: 5 7 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Read 7 Done Done Done false 1004 Acked Write 1 Done Done Done false 1005 Acked Read 9 Done Done Done false 1006 Acked Write 5 Done Done Done false 1007 Acked Read 1 Done Done Done false 1008 Acked Read 2 Done Done Done false 1009 Acked Read 6 Done Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Read 5 Sent Done Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 20 1 2 23 Done 10 29 28 67 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:14.699279682Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1457} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 79 79 79 149 149 149 F F F 3 060-079 75 75 75 139 139 139 F F F 4 080-099 78 78 78 147 147 147 F F F 5 100-119 78 78 78 148 148 148 F F F 6 120-139 78 78 78 147 147 147 F T T <--- 7 140-159 74 74 74 137 137 137 F F F 8 160-179 76 80 80 141 150 150 T F F <--- 9 180-199 69 80 80 131 150 150 T F F <--- 10 200-219 65 65 65 124 124 124 F F F 11 220-239 68 80 80 128 150 150 T F F <--- 12 240-259 78 78 78 148 148 148 F T T <--- 13 260-279 75 75 75 139 139 139 F F F 14 280-299 76 76 76 145 145 145 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 71 71 71 134 134 134 F F F 17 340-359 79 80 80 149 150 150 T T T <--- 18 360-379 76 76 76 142 142 142 F T T <--- 19 380-399 78 80 80 148 151 151 F F F <--- 20 400-419 70 70 70 133 133 133 F T T <--- 21 420-439 78 78 78 148 148 148 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 68 80 80 128 150 150 T F F <--- 24 480-499 66 66 66 125 125 125 F F F 25 500-519 78 78 78 148 148 148 F F F 26 520-539 78 78 78 148 148 148 F F F 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 80, Max flush: 151 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 79 79 79 149 149 149 F F F 3 060-079 75 75 75 139 139 139 F F F 4 080-099 78 78 78 147 147 147 F F F 5 100-119 78 78 78 148 148 148 F F F 6 120-139 81 81 81 152 152 152 F F F 7 140-159 74 74 74 137 137 137 F F F 8 160-179 80 80 80 150 150 150 F F F 9 180-199 80 80 80 150 150 150 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 80 80 80 150 150 150 F F F 12 240-259 81 81 81 152 152 152 F F F 13 260-279 75 75 75 139 139 139 F F F 14 280-299 76 76 76 145 145 145 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 71 71 71 134 134 134 F F F 17 340-359 81 81 81 152 152 152 F F F 18 360-379 81 81 81 152 152 152 F F F 19 380-399 80 80 80 151 151 151 F F F 20 400-419 81 81 81 152 152 152 F F F 21 420-439 78 78 78 148 148 148 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 80 80 80 150 150 150 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 78 78 78 148 148 148 F F F 26 520-539 78 78 78 148 148 148 F F F 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 75 75 75 139 139 139 F F F Max gen: 81, Max flush: 152 Begin loop 41 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.948165466Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.956462753Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.95675938Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.956785836Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.956794525Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.956806083Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465} {"msg":"Crucible 17328d87-e79f-4671-aca8-5a1f88fe09bb has session id: 88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957132176Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Upstairs opts: Upstairs UUID: 17328d87-e79f-4671-aca8-5a1f88fe09bb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957161321Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Crucible stats registered with UUID: 17328d87-e79f-4671-aca8-5a1f88fe09bb","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.95717242Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:18.957181958Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957328657Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"io task","client":"0","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957352133Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"io task","client":"2","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957362312Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"io task","client":"1","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957476325Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"task":"crutest"} {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957500722Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465} {"msg":"17328d87-e79f-4671-aca8-5a1f88fe09bb active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957524028Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957533307Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"client":"0","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957544805Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"client":"1","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957554594Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"client":"2","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957564053Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"io task","client":"2","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957573921Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"io task","client":"0","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"ds_connection connected from Ok(127.0.0.1:46162)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957603667Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"io task","client":"2","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957618715Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"io task","client":"1","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"ds_connection connected from Ok(127.0.0.1:43760)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957628983Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"io task","client":"0","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"ds_connection connected from Ok(127.0.0.1:46811)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.957730868Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"io task","client":"1","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958213518Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"client":"0","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958315903Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"client":"2","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958330341Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"client":"1","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958523982Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958628737Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"[0]R flush_numbers[0..12]: [135, 139, 149, 139, 147, 148, 152, 137, 150, 150, 124, 150]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.95867578Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"[0]R generation[0..12]: [72, 75, 79, 75, 78, 78, 81, 74, 80, 80, 65, 80]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958687908Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958701346Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"[1]R flush_numbers[0..12]: [135, 139, 149, 139, 147, 148, 152, 137, 150, 150, 124, 150]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958712025Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"[1]R generation[0..12]: [72, 75, 79, 75, 78, 78, 81, 74, 80, 80, 65, 80]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958722283Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958733182Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"[2]R flush_numbers[0..12]: [135, 139, 149, 139, 147, 148, 152, 137, 150, 150, 124, 150]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.95874368Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"[2]R generation[0..12]: [72, 75, 79, 75, 78, 78, 81, 74, 80, 80, 65, 80]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958753849Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958764657Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958773896Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958783514Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Next flush: 153","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958792683Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958801932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"":"downstairs","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.95881172Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958820459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"client":"0","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958830317Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"client":"1","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958839966Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"client":"2","":"downstairs","session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958850984Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958860103Z",Wait for a query_work_queue command to finish before sending IO "hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"17328d87-e79f-4671-aca8-5a1f88fe09bb is now active with session: 88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958873961Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.95888278Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958890289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465,"session_id":"88a364d1-db4c-4ef1-ab3a-b09e48f9e9c5"} {"msg":"Activated sub_volume 17328d87-e79f-4671-aca8-5a1f88fe09bb","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:18.958899367Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 583 len 20480 data: 5 8 10 7 9 02/30 Read block 241 len 36864 03/30 Read block 239 len 24576 04/30 Read block 307 len 40960 05/30 Flush 06/30 Write block 396 len 36864 data: 7 7 8 6 6 4 5 5 5 07/30 Read block 478 len 36864 08/30 Read block 133 len 20480 09/30 Read block 18 len 20480 10/30 Write block 47 len 16384 data: 9 8 8 6 11/30 Read block 305 len 40960 12/30 Read block 344 len 12288 13/30 Read block 313 len 24576 14/30 Flush 15/30 Write block 138 len 28672 data: 5 8 11 12 10 11 11 16/30 Write block 151 len 20480 data: 5 5 4 3 3 17/30 Write block 85 len 20480 data: 9 8 10 9 10 18/30 Write block 241 len 16384 data: 10 9 9 8 19/30 Read block 136 len 12288 20/30 Write block 323 len 24576 data: 6 6 6 6 9 8 21/30 Write block 221 len 12288 data: 4 5 8 22/30 Write block 269 len 16384 data: 9 6 7 7 23/30 Read block 333 len 4096 24/30 Read block 446 len 28672 25/30 Write block 388 len 32768 data: 7 7 8 11 10 10 8 8 26/30 Read block 378 len 24576 27/30 Read block 166 len 4096 28/30 Write block 108 len 12288 data: 8 8 7 29/30 Read block 466 len 12288 30/30 Read block 257 len 32768 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 9 Done Sent Done false 1006 Acked Read 9 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 5 Done Sent Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Read 10 Done Sent Done false 1011 Acked Read 3 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Write 6 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Read 1 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Write 8 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Read 8 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 0 25 Done 25 0 25 50 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) Some(JobId(1004)) Some(JobId(1013)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:18.968306712Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1465} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 82 79 82 154 149 154 F F F <--- 3 060-079 75 75 75 139 139 139 F F F 4 080-099 78 78 78 147 147 147 T F T <--- 5 100-119 78 78 78 148 148 148 T F T <--- 6 120-139 81 81 81 152 152 152 T F T <--- 7 140-159 74 74 74 137 137 137 T F T <--- 8 160-179 80 80 80 150 150 150 F F F 9 180-199 80 80 80 150 150 150 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 80 80 80 150 150 150 T F T <--- 12 240-259 81 81 81 152 152 152 T F T <--- 13 260-279 75 75 75 139 139 139 T F T <--- 14 280-299 76 76 76 145 145 145 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 71 71 71 134 134 134 T F T <--- 17 340-359 81 81 81 152 152 152 F F F 18 360-379 81 81 81 152 152 152 F F F 19 380-399 82 80 82 154 151 154 T F T <--- 20 400-419 82 81 82 154 152 154 F F F <--- 21 420-439 78 78 78 148 148 148 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 80 80 80 150 150 150 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 78 78 78 148 148 148 F F F 26 520-539 78 78 78 148 148 148 F F F 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 82 82 82 153 153 153 F F F Max gen: 82, Max flush: 154 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 82 82 82 154 154 154 F F F 3 060-079 75 75 75 139 139 139 F F F 4 080-099 83 83 83 155 155 155 F F F 5 100-119 83 83 83 155 155 155 F F F 6 120-139 83 83 83 155 155 155 F F F 7 140-159 83 83 83 155 155 155 F F F 8 160-179 80 80 80 150 150 150 F F F 9 180-199 80 80 80 150 150 150 F F F 10 200-219 65 65 65 124 124 124 F F F 11 220-239 83 83 83 155 155 155 F F F 12 240-259 83 83 83 155 155 155 F F F 13 260-279 83 83 83 155 155 155 F F F 14 280-299 76 76 76 145 145 145 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 83 83 83 155 155 155 F F F 17 340-359 81 81 81 152 152 152 F F F 18 360-379 81 81 81 152 152 152 F F F 19 380-399 83 83 83 155 155 155 F F F 20 400-419 82 82 82 154 154 154 F F F 21 420-439 78 78 78 148 148 148 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 80 80 80 150 150 150 F F F 24 480-499 66 66 66 125 125 125 F F F 25 500-519 78 78 78 148 148 148 F F F 26 520-539 78 78 78 148 148 148 F F F 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 82 82 82 153 153 153 F F F Max gen: 83, Max flush: 155 Begin loop 42 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.214591027Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:23.221816821Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.221881102Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.229195782Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.229458804Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.22948573Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.229494429Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.229505987Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473} {"msg":"Crucible 37d6bd36-cf8e-4f46-84df-74804e54a56f has session id: a9659c51-1ffa-44af-abf9-59621213b09b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.229775048Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Upstairs opts: Upstairs UUID: 37d6bd36-cf8e-4f46-84df-74804e54a56f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.22982972Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Crucible stats registered with UUID: 37d6bd36-cf8e-4f46-84df-74804e54a56f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.229842498Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:23.229851557Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.229987427Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"io task","client":"0","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230009973Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"io task","client":"1","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230019882Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"io task","client":"2","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230152603Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"task":"crutest"} {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.23016918Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473} {"msg":"37d6bd36-cf8e-4f46-84df-74804e54a56f active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230192337Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230201575Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"client":"0","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230211274Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"client":"1","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230220793Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"client":"2","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230230301Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"io task","client":"2","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.23024014Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"io task","client":"0","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230250098Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"io task","client":"1","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"ds_connection connected from Ok(127.0.0.1:51958)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230313229Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"io task","client":"2","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"ds_connection connected from Ok(127.0.0.1:50632)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230340835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"io task","client":"1","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"ds_connection connected from Ok(127.0.0.1:41455)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230352743Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"io task","client":"0","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230872827Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"client":"2","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.230953855Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"client":"1","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.23105862Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"client":"0","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231162215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231200829Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"[0]R flush_numbers[0..12]: [135, 139, 154, 139, 155, 155, 155, 155, 150, 150, 124, 155]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231365775Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"[0]R generation[0..12]: [72, 75, 82, 75, 83, 83, 83, 83, 80, 80, 65, 83]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231379383Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231390692Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"[1]R flush_numbers[0..12]: [135, 139, 154, 139, 155, 155, 155, 155, 150, 150, 124, 155]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.23140142Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"[1]R generation[0..12]: [72, 75, 82, 75, 83, 83, 83, 83, 80, 80, 65, 83]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231411689Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231422387Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"[2]R flush_numbers[0..12]: [135, 139, 154, 139, 155, 155, 155, 155, 150, 150, 124, 155]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231432905Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"[2]R generation[0..12]: [72, 75, 82, 75, 83, 83, 83, 83, 80, 80, 65, 83]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231445104Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231455992Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Max found gen is 84","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231465281Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Generation requested: 84 >= found:84","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231474749Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Next flush: 156","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231483898Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231493137Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"":"downstairs","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231503125Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231511894Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"client":"0","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231521742Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"client":"1","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231531491Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"client":"2","":"downstairs","session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:13:23.23154138Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231553508Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"37d6bd36-cf8e-4f46-84df-74804e54a56f is now active with session: a9659c51-1ffa-44af-abf9-59621213b09b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231563236Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231572245Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231579854Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473,"session_id":"a9659c51-1ffa-44af-abf9-59621213b09b"} {"msg":"Activated sub_volume 37d6bd36-cf8e-4f46-84df-74804e54a56f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:23.231588793Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 531 len 28672 data: 10 12 13 12 13 11 11 02/30 Read block 137 len 24576 03/30 Write block 187 len 36864 data: 4 3 4 6 7 8 8 8 8 04/30 Read block 355 len 40960 05/30 Read block 248 len 28672 06/30 Write block 118 len 36864 data: 8 7 9 8 9 7 5 4 5 07/30 Read block 160 len 32768 08/30 Write block 414 len 8192 data: 4 4 09/30 Read block 589 len 4096 10/30 Read block 244 len 12288 11/30 Read block 156 len 16384 12/30 Read block 416 len 12288 13/30 Write block 140 len 16384 data: 12 13 11 12 14/30 Write block 107 len 32768 data: 10 9 9 8 7 5 5 7 15/30 Write block 109 len 16384 data: 10 9 8 6 16/30 Write block 481 len 20480 data: 5 5 5 3 2 17/30 Write block 272 len 40960 data: 8 8 7 5 5 5 3 8 8 8 18/30 Read block 264 len 4096 19/30 Read block 305 len 16384 20/30 Read block 398 len 28672 21/30 Read block 98 len 28672 22/30 Read block 368 len 36864 23/30 Read block 8 len 20480 24/30 Flush 25/30 Write block 222 len 40960 data: 6 9 9 8 10 10 11 11 10 6 26/30 Write block 193 len 32768 data: 9 9 9 8 9 8 8 7 27/30 Read block 528 len 36864 28/30 Read block 340 len 4096 29/30 Write block 124 len 12288 data: 6 5 6 30/30 Read block 246 len 16384 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Write 9 Done Done Done false 1003 Acked Read 10 Done Done Done false 1004 Acked Read 7 Done Done Done false 1005 Acked Write 9 Done Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Read 3 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Read 5 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Write 3 Sent Done Done false 1029 Acked Read 4 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 0 0 24 Done 6 30 30 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:23.240927578Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1473} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 82 82 82 154 154 154 F F F 3 060-079 75 75 75 139 139 139 F F F 4 080-099 83 83 83 155 155 155 F F F 5 100-119 83 84 84 155 156 156 T F F <--- 6 120-139 83 84 84 155 156 156 T T T <--- 7 140-159 83 84 84 155 156 156 F F F <--- 8 160-179 80 80 80 150 150 150 F F F 9 180-199 80 84 84 150 156 156 T T T <--- 10 200-219 65 65 65 124 124 124 F T T <--- 11 220-239 83 83 83 155 155 155 F T T <--- 12 240-259 83 83 83 155 155 155 F F F 13 260-279 83 84 84 155 156 156 F F F <--- 14 280-299 76 84 84 145 156 156 F F F <--- 15 300-319 74 74 74 136 136 136 F F F 16 320-339 83 83 83 155 155 155 F F F 17 340-359 81 81 81 152 152 152 F F F 18 360-379 81 81 81 152 152 152 F F F 19 380-399 83 83 83 155 155 155 F F F 20 400-419 82 84 84 154 156 156 F F F <--- 21 420-439 78 78 78 148 148 148 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 80 80 80 150 150 150 F F F 24 480-499 66 84 84 125 156 156 F F F <--- 25 500-519 78 78 78 148 148 148 F F F 26 520-539 78 84 84 148 156 156 T F F <--- 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 82 82 82 153 153 153 F F F Max gen: 84, Max flush: 156 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 135 135 135 F F F 1 020-039 75 75 75 139 139 139 F F F 2 040-059 82 82 82 154 154 154 F F F 3 060-079 75 75 75 139 139 139 F F F 4 080-099 83 83 83 155 155 155 F F F 5 100-119 84 84 84 156 156 156 F F F 6 120-139 85 85 85 157 157 157 F F F 7 140-159 84 84 84 156 156 156 F F F 8 160-179 80 80 80 150 150 150 F F F 9 180-199 85 85 85 157 157 157 F F F 10 200-219 85 85 85 157 157 157 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 83 83 83 155 155 155 F F F 13 260-279 84 84 84 156 156 156 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 83 83 83 155 155 155 F F F 17 340-359 81 81 81 152 152 152 F F F 18 360-379 81 81 81 152 152 152 F F F 19 380-399 83 83 83 155 155 155 F F F 20 400-419 84 84 84 156 156 156 F F F 21 420-439 78 78 78 148 148 148 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 80 80 80 150 150 150 F F F 24 480-499 84 84 84 156 156 156 F F F 25 500-519 78 78 78 148 148 148 F F F 26 520-539 84 84 84 156 156 156 F F F 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 82 82 82 153 153 153 F F F Max gen: 85, Max flush: 157 Begin loop 43 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.490208734Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:27.497385265Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.497447876Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.504324201Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.504592092Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.504619768Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.504628087Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.504639415Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481} {"msg":"Crucible ac361f86-c56c-48e0-9c87-36bf39177567 has session id: 3131257d-9343-4842-a330-c799425773e8","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.504972336Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Upstairs opts: Upstairs UUID: ac361f86-c56c-48e0-9c87-36bf39177567, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505010341Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Crucible stats registered with UUID: ac361f86-c56c-48e0-9c87-36bf39177567","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505021429Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:27.505030498Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.50515577Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"io task","client":"2","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505179226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"io task","client":"0","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505191055Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"io task","client":"1","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505315646Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"task":"crutest"} {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505339703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481} {"msg":"ac361f86-c56c-48e0-9c87-36bf39177567 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505364559Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505373928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"client":"0","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505383386Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"client":"1","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505392835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"client":"2","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505402354Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"io task","client":"2","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505412132Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"io task","client":"0","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505421951Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"io task","client":"1","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"ds_connection connected from Ok(127.0.0.1:57034)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505475273Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"io task","client":"2","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"ds_connection connected from Ok(127.0.0.1:43012)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505486021Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"io task","client":"1","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"ds_connection connected from Ok(127.0.0.1:44745)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.505567609Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"io task","client":"0","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506130637Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"client":"1","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506155314Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"client":"2","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506219764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"client":"0","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506427554Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.50645434Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"[0]R flush_numbers[0..12]: [135, 139, 154, 139, 155, 156, 157, 156, 150, 157, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506482986Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"[0]R generation[0..12]: [72, 75, 82, 75, 83, 84, 85, 84, 80, 85, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506494014Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506505042Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"[1]R flush_numbers[0..12]: [135, 139, 154, 139, 155, 156, 157, 156, 150, 157, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506515761Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"[1]R generation[0..12]: [72, 75, 82, 75, 83, 84, 85, 84, 80, 85, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506526279Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506537198Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"[2]R flush_numbers[0..12]: [135, 139, 154, 139, 155, 156, 157, 156, 150, 157, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506547706Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"[2]R generation[0..12]: [72, 75, 82, 75, 83, 84, 85, 84, 80, 85, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506557915Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506568693Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Max found gen is 86","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506578062Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Generation requested: 86 >= found:86","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.50658778Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Next flush: 158","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506597059Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506607827Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"":"downstairs","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506617776Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506626525Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"client":"0","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506641992Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"client":"1","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506651821Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"client":"2","":"downstairs","session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.50666186Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506671028Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Wait for a query_work_queue command to finish before sending IO ac361f86-c56c-48e0-9c87-36bf39177567 is now active with session: 3131257d-9343-4842-a330-c799425773e8","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506683546Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506692615Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506700054Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481,"session_id":"3131257d-9343-4842-a330-c799425773e8"} {"msg":"Activated sub_volume ac361f86-c56c-48e0-9c87-36bf39177567","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:27.506709093Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 264 len 40960 data: 12 12 12 12 10 10 7 8 9 9 02/30 Write block 371 len 12288 data: 8 7 8 03/30 Write block 517 len 28672 data: 8 7 7 6 7 7 6 04/30 Read block 537 len 32768 05/30 Read block 239 len 24576 06/30 Read block 43 len 8192 07/30 Read block 575 len 16384 08/30 Write block 17 len 24576 data: 6 5 5 4 4 3 09/30 Read block 406 len 36864 10/30 Read block 213 len 24576 11/30 Read block 590 len 8192 12/30 Write block 139 len 40960 data: 9 13 14 12 13 12 12 9 8 6 13/30 Flush 14/30 Write block 468 len 32768 data: 8 8 8 7 7 8 9 7 15/30 Write block 358 len 12288 data: 9 7 9 16/30 Read block 590 len 12288 17/30 Write block 190 len 16384 data: 7 8 9 10 18/30 Read block 510 len 8192 19/30 Write block 43 len 8192 data: 9 8 20/30 Write block 311 len 36864 data: 9 7 8 8 9 12 12 12 11 21/30 Flush 22/30 Write block 17 len 20480 data: 7 6 6 5 5 23/30 Read block 108 len 4096 24/30 Read block 323 len 8192 25/30 Read block 476 len 12288 26/30 Read block 377 len 32768 27/30 Write block 331 len 28672 data: 10 10 8 9 8 8 6 28/30 Read block 547 len 24576 29/30 Write block 364 len 12288 data: 4 5 5 30/30 Write block 40 len 40960 data: 8 8 9 10 9 11 10 10 9 9 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Write 7 Sent Done Done false 1003 Acked Read 8 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 2 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Read 2 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Read 2 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Write 3 Sent Done Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 2 31 Done 2 29 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:27.515836289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1481} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 86 86 135 158 158 F T T <--- 1 020-039 75 86 86 139 158 158 F T T <--- 2 040-059 82 86 86 154 159 159 F T T <--- 3 060-079 75 75 75 139 139 139 F F F 4 080-099 83 83 83 155 155 155 F F F 5 100-119 84 84 84 156 156 156 F F F 6 120-139 85 86 86 157 158 158 F F F <--- 7 140-159 84 86 86 156 158 158 F F F <--- 8 160-179 80 80 80 150 150 150 F F F 9 180-199 85 86 86 157 159 159 F F F <--- 10 200-219 85 85 85 157 157 157 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 83 83 83 155 155 155 F F F 13 260-279 84 86 86 156 158 158 T F F <--- 14 280-299 84 84 84 156 156 156 F F F 15 300-319 74 86 86 136 159 159 F F F <--- 16 320-339 83 83 83 155 155 155 F T T <--- 17 340-359 81 86 86 152 159 159 F F F <--- 18 360-379 81 86 86 152 159 159 T T T <--- 19 380-399 83 83 83 155 155 155 F F F 20 400-419 84 84 84 156 156 156 F F F 21 420-439 78 78 78 148 148 148 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 80 86 86 150 159 159 F F F <--- 24 480-499 84 84 84 156 156 156 F F F 25 500-519 78 86 86 148 158 158 F F F <--- 26 520-539 84 86 86 156 158 158 F F F <--- 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 82 82 82 153 153 153 F F F Max gen: 86, Max flush: 159 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 43 Verifying data now No change after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 87 87 87 160 160 160 F F F 1 020-039 87 87 87 160 160 160 F F F 2 040-059 87 87 87 160 160 160 F F F 3 060-079 75 75 75 139 139 139 F F F 4 080-099 83 83 83 155 155 155 F F F 5 100-119 84 84 84 156 156 156 F F F 6 120-139 86 86 86 158 158 158 F F F 7 140-159 86 86 86 158 158 158 F F F 8 160-179 80 80 80 150 150 150 F F F 9 180-199 86 86 86 159 159 159 F F F 10 200-219 85 85 85 157 157 157 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 83 83 83 155 155 155 F F F 13 260-279 86 86 86 158 158 158 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 86 86 86 159 159 159 F F F 16 320-339 87 87 87 160 160 160 F F F 17 340-359 86 86 86 159 159 159 F F F 18 360-379 87 87 87 160 160 160 F F F 19 380-399 83 83 83 155 155 155 F F F 20 400-419 84 84 84 156 156 156 F F F 21 420-439 78 78 78 148 148 148 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 86 86 86 159 159 159 F F F 24 480-499 84 84 84 156 156 156 F F F 25 500-519 86 86 86 158 158 158 F F F 26 520-539 86 86 86 158 158 158 F F F 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 82 82 82 153 153 153 F F F Max gen: 87, Max flush: 160 Begin loop 44 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.77637391Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.784631543Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.784918831Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.784945327Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.784953756Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.784967594Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489} {"msg":"Crucible f6666b68-3578-410a-a70f-bae9ce82d757 has session id: dfaeaac7-fb61-4f01-a92f-dd514e9210ec","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785326321Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Upstairs opts: Upstairs UUID: f6666b68-3578-410a-a70f-bae9ce82d757, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785355127Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Crucible stats registered with UUID: f6666b68-3578-410a-a70f-bae9ce82d757","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785366296Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:31.785375524Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.78553883Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"io task","client":"0","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785575355Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"io task","client":"1","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785586303Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"io task","client":"2","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785697057Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"task":"crutest"} {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785714895Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489} {"msg":"f6666b68-3578-410a-a70f-bae9ce82d757 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.78574918Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785758598Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"client":"0","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785768267Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"client":"1","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785777715Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"client":"2","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785787114Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"io task","client":"2","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785798922Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"io task","client":"1","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785849565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"io task","client":"0","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:47212)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785860383Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"io task","client":"2","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:60369)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785870592Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"io task","client":"1","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:41964)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.785943131Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"io task","client":"0","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786440049Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"client":"0","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786466645Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"client":"1","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786600575Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"client":"2","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786729216Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786746054Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"[0]R flush_numbers[0..12]: [160, 160, 160, 139, 155, 156, 158, 158, 150, 159, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786885924Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"[0]R generation[0..12]: [87, 87, 87, 75, 83, 84, 86, 86, 80, 86, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786897052Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.78690798Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"[1]R flush_numbers[0..12]: [160, 160, 160, 139, 155, 156, 158, 158, 150, 159, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786918719Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"[1]R generation[0..12]: [87, 87, 87, 75, 83, 84, 86, 86, 80, 86, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786930867Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786941865Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"[2]R flush_numbers[0..12]: [160, 160, 160, 139, 155, 156, 158, 158, 150, 159, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786952354Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"[2]R generation[0..12]: [87, 87, 87, 75, 83, 84, 86, 86, 80, 86, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786962662Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786973561Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Max found gen is 88","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786983039Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.786992718Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Next flush: 161","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.787002027Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.787011395Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"":"downstairs","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.787021074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.787029732Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"client":"0","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.787039391Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"client":"1","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.78704897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"client":"2","":"downstairs","session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.787058718Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.787067727Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"f6666b68-3578-410a-a70f-bae9ce82d757 is now active with session: dfaeaac7-fb61-4f01-a92f-dd514e9210ec","v":0,"name":"crucible","level"Wait for a query_work_queue command to finish before sending IO :30,"time":"2026-04-15T18:13:31.787083605Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.787092773Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.787100322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489,"session_id":"dfaeaac7-fb61-4f01-a92f-dd514e9210ec"} {"msg":"Activated sub_volume f6666b68-3578-410a-a70f-bae9ce82d757","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:31.787109321Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 340 len 12288 data: 8 10 11 02/30 Read block 2 len 24576 03/30 Flush 04/30 Write block 163 len 8192 data: 14 13 05/30 Write block 187 len 28672 data: 5 4 5 8 9 10 11 06/30 Read block 56 len 24576 07/30 Read block 0 len 32768 08/30 Read block 591 len 4096 09/30 Write block 414 len 28672 data: 5 5 4 5 6 6 5 10/30 Write block 46 len 32768 data: 11 11 10 10 7 6 7 6 11/30 Write block 110 len 8192 data: 10 9 12/30 Read block 116 len 12288 13/30 Write block 55 len 32768 data: 6 8 8 6 5 4 5 4 14/30 Read block 406 len 32768 15/30 Read block 203 len 28672 16/30 Read block 79 len 4096 17/30 Flush 18/30 Write block 174 len 36864 data: 7 6 9 10 11 11 11 9 10 19/30 Read block 10 len 8192 20/30 Read block 567 len 20480 21/30 Read block 596 len 16384 22/30 Read block 453 len 32768 23/30 Read block 39 len 40960 24/30 Read block 335 len 4096 25/30 Write block 306 len 36864 data: 8 4 4 7 8 10 8 9 9 26/30 Read block 58 len 32768 27/30 Write block 467 len 8192 data: 8 9 28/30 Read block 305 len 16384 29/30 Write block 178 len 36864 data: 12 12 12 10 11 9 8 7 7 30/30 Write block 306 len 16384 data: 9 5 5 8 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 6 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 2 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Write 8 Done Done Sent false 1010 Acked Write 2 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Read 4 Done Done Sent false 1021 Acked Read 8 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Read 8 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 29 33 Done 28 28 1 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:31.796415391Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1489} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 87 87 87 160 160 160 F F F 1 020-039 87 87 87 160 160 160 F F F 2 040-059 88 88 87 162 162 160 F F F <--- 3 060-079 88 88 75 162 162 139 F F F <--- 4 080-099 83 83 83 155 155 155 F F F 5 100-119 88 88 84 162 162 156 F F F <--- 6 120-139 86 86 86 158 158 158 F F F 7 140-159 86 86 86 158 158 158 F F F 8 160-179 88 88 80 162 162 150 T T F <--- 9 180-199 88 88 86 162 162 159 T T F <--- 10 200-219 85 85 85 157 157 157 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 83 83 83 155 155 155 F F F 13 260-279 86 86 86 158 158 158 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 86 86 86 159 159 159 T T F <--- 16 320-339 87 87 87 160 160 160 F F F 17 340-359 88 88 86 161 161 159 F F T <--- 18 360-379 87 87 87 160 160 160 F F F 19 380-399 83 83 83 155 155 155 F F F 20 400-419 88 88 84 162 162 156 F F F <--- 21 420-439 88 88 78 162 162 148 F F F <--- 22 440-459 76 76 76 143 143 143 F F F 23 460-479 86 86 86 159 159 159 T T F <--- 24 480-499 84 84 84 156 156 156 F F F 25 500-519 86 86 86 158 158 158 F F F 26 520-539 86 86 86 158 158 158 F F F 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 82 82 82 153 153 153 F F F Max gen: 88, Max flush: 162 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 87 87 87 160 160 160 F F F 1 020-039 87 87 87 160 160 160 F F F 2 040-059 88 88 88 162 162 162 F F F 3 060-079 88 88 88 162 162 162 F F F 4 080-099 83 83 83 155 155 155 F F F 5 100-119 88 88 88 162 162 162 F F F 6 120-139 86 86 86 158 158 158 F F F 7 140-159 86 86 86 158 158 158 F F F 8 160-179 89 89 89 163 163 163 F F F 9 180-199 89 89 89 163 163 163 F F F 10 200-219 85 85 85 157 157 157 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 83 83 83 155 155 155 F F F 13 260-279 86 86 86 158 158 158 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 89 89 89 163 163 163 F F F 16 320-339 87 87 87 160 160 160 F F F 17 340-359 88 88 88 161 161 161 F F F 18 360-379 87 87 87 160 160 160 F F F 19 380-399 83 83 83 155 155 155 F F F 20 400-419 88 88 88 162 162 162 F F F 21 420-439 88 88 88 162 162 162 F F F 22 440-459 76 76 76 143 143 143 F F F 23 460-479 89 89 89 163 163 163 F F F 24 480-499 84 84 84 156 156 156 F F F 25 500-519 86 86 86 158 158 158 F F F 26 520-539 86 86 86 158 158 158 F F F 27 540-559 79 79 79 149 149 149 F F F 28 560-579 79 79 79 149 149 149 F F F 29 580-599 82 82 82 153 153 153 F F F Max gen: 89, Max flush: 163 Begin loop 45 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.040616709Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.048689469Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.048994505Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049020751Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.04902959Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049041438Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497} {"msg":"Crucible ddaaa5d8-6282-442a-9c1b-43880a744d6d has session id: c0c2f843-fb38-4036-8a00-a4bebf963cba","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049394966Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Upstairs opts: Upstairs UUID: ddaaa5d8-6282-442a-9c1b-43880a744d6d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049426642Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Crucible stats registered with UUID: ddaaa5d8-6282-442a-9c1b-43880a744d6d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.04943705Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:36.049446209Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049593197Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"io task","client":"0","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049617274Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"io task","client":"2","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049627342Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"io task","client":"1","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049735107Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"task":"crutest"} {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049751834Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497} {"msg":"ddaaa5d8-6282-442a-9c1b-43880a744d6d active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049794188Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049803576Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"client":"0","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049813055Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"client":"1","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049822454Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"client":"2","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049831832Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"io task","client":"2","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049841671Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"io task","client":"0","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049854069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"io task","client":"1","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"ds_connection connected from Ok(127.0.0.1:37493)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049921949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"io task","client":"0","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"ds_connection connected from Ok(127.0.0.1:58437)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049947665Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"io task","client":"1","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"ds_connection connected from Ok(127.0.0.1:44564)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.049958784Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"io task","client":"2","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.050644304Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"client":"2","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.050662821Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"client":"1","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.050675569Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"client":"0","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.050981184Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051029047Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"[0]R flush_numbers[0..12]: [160, 160, 162, 162, 155, 162, 158, 158, 163, 163, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051089389Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"[0]R generation[0..12]: [87, 87, 88, 88, 83, 88, 86, 86, 89, 89, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051100327Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051111115Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"[1]R flush_numbers[0..12]: [160, 160, 162, 162, 155, 162, 158, 158, 163, 163, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051121704Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"[1]R generation[0..12]: [87, 87, 88, 88, 83, 88, 86, 86, 89, 89, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051131802Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051142621Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"[2]R flush_numbers[0..12]: [160, 160, 162, 162, 155, 162, 158, 158, 163, 163, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051152989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"[2]R generation[0..12]: [87, 87, 88, 88, 83, 88, 86, 86, 89, 89, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051163098Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051173906Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051183095Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051192703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Next flush: 164","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051202922Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051212201Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"":"downstairs","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051222079Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051230808Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"client":"0","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051240556Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"client":"1","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051254724Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"client":"2","":"downstairs","session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051264623Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051273782Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"ddaaa5d8-6282-442a-9c1b-43880a744d6d is now active with session: c0c2f843-fb38-4036-8a00-a4bebf963cba","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.05128349Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051292569Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051300128Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497,"session_id":"c0c2f843-fb38-4036-8a00-a4bebf963cba"} {"msg":"Activated sub_volume ddaaa5d8-6282-442a-9c1b-43880a744d6d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:36.051309186Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 23 len 16384 data: 4 6 7 6 02/30 Write block 192 len 4096 data: 11 03/30 Read block 34 len 28672 04/30 Write block 2 len 40960 data: 5 5 7 7 7 7 7 7 8 8 05/30 Read block 292 len 4096 06/30 Flush 07/30 Write block 500 len 4096 data: 8 08/30 Read block 239 len 28672 09/30 Read block 366 len 24576 10/30 Read block 110 len 40960 11/30 Read block 178 len 20480 12/30 Write block 117 len 4096 data: 9 13/30 Flush 14/30 Write block 555 len 28672 data: 9 9 10 9 7 8 8 15/30 Write block 454 len 40960 data: 8 9 11 11 12 11 9 7 5 6 16/30 Read block 589 len 20480 17/30 Write block 458 len 28672 data: 13 12 10 8 6 7 4 18/30 Read block 15 len 12288 19/30 Read block 475 len 8192 20/30 Flush 21/30 Write block 266 len 24576 data: 13 13 11 11 8 9 22/30 Read block 171 len 12288 23/30 Write block 505 len 32768 data: 5 6 6 9 10 8 7 7 24/30 Write block 387 len 12288 data: 6 8 8 25/30 Read block 526 len 8192 26/30 Flush 27/30 Write block 55 len 20480 data: 7 9 9 7 6 28/30 Read block 559 len 36864 29/30 Read block 464 len 16384 30/30 Read block 492 len 24576 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Sent Done false 1001 Acked Write 1 Done Sent Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Write 10 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 7 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 7 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Read 5 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Read 3 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Read 6 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 0 31 Done 29 0 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:36.060603088Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1497} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 87 90 164 160 164 F F F <--- 1 020-039 90 87 90 164 160 164 F F F <--- 2 040-059 88 88 88 162 162 162 T F T <--- 3 060-079 88 88 88 162 162 162 F F F 4 080-099 83 83 83 155 155 155 F F F 5 100-119 90 88 90 165 162 165 F F F <--- 6 120-139 86 86 86 158 158 158 F F F 7 140-159 86 86 86 158 158 158 F F F 8 160-179 89 89 89 163 163 163 F F F 9 180-199 90 89 90 164 163 164 F F F <--- 10 200-219 85 85 85 157 157 157 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 83 83 83 155 155 155 F F F 13 260-279 90 86 90 167 158 167 F F F <--- 14 280-299 84 84 84 156 156 156 F F F 15 300-319 89 89 89 163 163 163 F F F 16 320-339 87 87 87 160 160 160 F F F 17 340-359 88 88 88 161 161 161 F F F 18 360-379 87 87 87 160 160 160 F F F 19 380-399 90 83 90 167 155 167 F F F <--- 20 400-419 88 88 88 162 162 162 F F F 21 420-439 88 88 88 162 162 162 F F F 22 440-459 90 76 90 166 143 166 F F F <--- 23 460-479 90 89 90 166 163 166 F F F <--- 24 480-499 84 84 84 156 156 156 F F F 25 500-519 90 86 90 167 158 167 F F F <--- 26 520-539 86 86 86 158 158 158 F F F 27 540-559 90 79 90 166 149 166 F F F <--- 28 560-579 90 79 90 166 149 166 F F F <--- 29 580-599 82 82 82 153 153 153 F F F Max gen: 90, Max flush: 167 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 164 164 164 F F F 1 020-039 90 90 90 164 164 164 F F F 2 040-059 91 91 91 168 168 168 F F F 3 060-079 88 88 88 162 162 162 F F F 4 080-099 83 83 83 155 155 155 F F F 5 100-119 90 90 90 165 165 165 F F F 6 120-139 86 86 86 158 158 158 F F F 7 140-159 86 86 86 158 158 158 F F F 8 160-179 89 89 89 163 163 163 F F F 9 180-199 90 90 90 164 164 164 F F F 10 200-219 85 85 85 157 157 157 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 83 83 83 155 155 155 F F F 13 260-279 90 90 90 167 167 167 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 89 89 89 163 163 163 F F F 16 320-339 87 87 87 160 160 160 F F F 17 340-359 88 88 88 161 161 161 F F F 18 360-379 87 87 87 160 160 160 F F F 19 380-399 90 90 90 167 167 167 F F F 20 400-419 88 88 88 162 162 162 F F F 21 420-439 88 88 88 162 162 162 F F F 22 440-459 90 90 90 166 166 166 F F F 23 460-479 90 90 90 166 166 166 F F F 24 480-499 84 84 84 156 156 156 F F F 25 500-519 90 90 90 167 167 167 F F F 26 520-539 86 86 86 158 158 158 F F F 27 540-559 90 90 90 166 166 166 F F F 28 560-579 90 90 90 166 166 166 F F F 29 580-599 82 82 82 153 153 153 F F F Max gen: 91, Max flush: 168 Begin loop 46 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.303383115Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.311580577Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.311870625Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.311898241Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.311908869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.311920297Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505} {"msg":"Crucible f5b895bd-8d2b-42f2-9604-715ec628da23 has session id: 72613417-2eb6-49a1-877b-a95c8a3a547a","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312272016Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Upstairs opts: Upstairs UUID: f5b895bd-8d2b-42f2-9604-715ec628da23, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312307261Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Crucible stats registered with UUID: f5b895bd-8d2b-42f2-9604-715ec628da23","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312318699Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:40.312328128Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312491764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"io task","client":"2","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312527709Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"io task","client":"0","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312538667Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"io task","client":"1","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312644081Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"task":"crutest"} {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312668578Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505} {"msg":"f5b895bd-8d2b-42f2-9604-715ec628da23 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312696424Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312705792Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"client":"0","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312715421Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"client":"1","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.31272508Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"client":"2","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312736738Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"io task","client":"2","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312746826Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"io task","client":"0","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312756755Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"io task","client":"1","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"ds_connection connected from Ok(127.0.0.1:61043)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312837113Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"io task","client":"0","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"ds_connection connected from Ok(127.0.0.1:50917)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312879517Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"io task","client":"1","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"ds_connection connected from Ok(127.0.0.1:60854)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.312901164Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"io task","client":"2","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313506665Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"client":"2","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313529442Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"client":"0","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313562377Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"client":"1","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313826499Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313836227Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"[0]R flush_numbers[0..12]: [164, 164, 168, 162, 155, 165, 158, 158, 163, 164, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313872252Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"[0]R generation[0..12]: [90, 90, 91, 88, 83, 90, 86, 86, 89, 90, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.31388331Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313894509Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"[1]R flush_numbers[0..12]: [164, 164, 168, 162, 155, 165, 158, 158, 163, 164, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313905107Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"[1]R generation[0..12]: [90, 90, 91, 88, 83, 90, 86, 86, 89, 90, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313918215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313929054Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"[2]R flush_numbers[0..12]: [164, 164, 168, 162, 155, 165, 158, 158, 163, 164, 157, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313939662Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"[2]R generation[0..12]: [90, 90, 91, 88, 83, 90, 86, 86, 89, 90, 85, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313949821Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313960539Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313969718Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313979166Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Next flush: 169","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313988335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.313997794Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"":"downstairs","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.314007442Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.314016191Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"client":"0","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.31402575Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"client":"1","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-04-15T18:13:40.314035208Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"client":"2","":"downstairs","session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.314048256Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.314057275Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"f5b895bd-8d2b-42f2-9604-715ec628da23 is now active with session: 72613417-2eb6-49a1-877b-a95c8a3a547a","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.314068233Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.314077132Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.314084611Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505,"session_id":"72613417-2eb6-49a1-877b-a95c8a3a547a"} {"msg":"Activated sub_volume f5b895bd-8d2b-42f2-9604-715ec628da23","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:40.31409355Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 481 len 24576 data: 6 6 6 4 3 2 03/30 Read block 493 len 28672 04/30 Write block 468 len 24576 data: 10 9 9 8 8 9 05/30 Write block 143 len 36864 data: 14 13 13 10 9 7 6 5 6 06/30 Read block 255 len 20480 07/30 Read block 428 len 12288 08/30 Write block 341 len 20480 data: 11 12 10 9 11 09/30 Write block 556 len 40960 data: 10 11 10 8 9 9 8 8 10 8 10/30 Read block 195 len 12288 11/30 Write block 58 len 32768 data: 8 7 5 6 5 5 7 7 12/30 Write block 213 len 4096 data: 4 13/30 Write block 173 len 24576 data: 6 8 7 10 11 13 14/30 Write block 117 len 4096 data: 10 15/30 Write block 452 len 16384 data: 5 6 9 10 16/30 Read block 76 len 32768 17/30 Read block 497 len 24576 18/30 Read block 198 len 32768 19/30 Read block 292 len 8192 20/30 Read block 210 len 20480 21/30 Read block 202 len 24576 22/30 Read block 581 len 12288 23/30 Read block 421 len 20480 24/30 Read block 210 len 20480 25/30 Read block 404 len 12288 26/30 Read block 532 len 28672 27/30 Write block 180 len 36864 data: 13 11 12 10 9 8 8 6 5 28/30 Write block 81 len 4096 data: 7 29/30 Write block 482 len 24576 data: 7 7 5 4 3 2 30/30 Read block 65 len 4096 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Done Sent Done false 1001 Acked Write 6 Done Sent Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Write 6 Done Sent Done false 1004 Acked Write 9 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Write 10 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Write 4 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Read 8 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Read 3 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Write 6 Done Sent Sent false 1029 Acked Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 2 32 Done 30 0 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) None Some(JobId(1000)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:40.322526547Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1505} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 164 164 164 F F F 1 020-039 90 90 90 164 164 164 F F F 2 040-059 91 91 91 168 168 168 T F T <--- 3 060-079 88 88 88 162 162 162 T F T <--- 4 080-099 83 83 83 155 155 155 T F T <--- 5 100-119 90 90 90 165 165 165 T F T <--- 6 120-139 86 86 86 158 158 158 F F F 7 140-159 86 86 86 158 158 158 T F T <--- 8 160-179 89 89 89 163 163 163 T F T <--- 9 180-199 90 90 90 164 164 164 T F T <--- 10 200-219 85 85 85 157 157 157 T F T <--- 11 220-239 85 85 85 157 157 157 F F F 12 240-259 83 83 83 155 155 155 F F F 13 260-279 90 90 90 167 167 167 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 89 89 89 163 163 163 F F F 16 320-339 87 87 87 160 160 160 F F F 17 340-359 88 88 88 161 161 161 T F T <--- 18 360-379 87 87 87 160 160 160 F F F 19 380-399 90 90 90 167 167 167 F F F 20 400-419 88 88 88 162 162 162 F F F 21 420-439 88 88 88 162 162 162 F F F 22 440-459 90 90 90 166 166 166 T F T <--- 23 460-479 90 90 90 166 166 166 T F T <--- 24 480-499 84 84 84 156 156 156 T F T <--- 25 500-519 90 90 90 167 167 167 F F F 26 520-539 86 86 86 158 158 158 F F F 27 540-559 90 90 90 166 166 166 T F T <--- 28 560-579 90 90 90 166 166 166 T F T <--- 29 580-599 82 82 82 153 153 153 F F F Max gen: 91, Max flush: 168 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 164 164 164 F F F 1 020-039 90 90 90 164 164 164 F F F 2 040-059 92 92 92 169 169 169 F F F 3 060-079 92 92 92 169 169 169 F F F 4 080-099 92 92 92 169 169 169 F F F 5 100-119 92 92 92 169 169 169 F F F 6 120-139 86 86 86 158 158 158 F F F 7 140-159 92 92 92 169 169 169 F F F 8 160-179 92 92 92 169 169 169 F F F 9 180-199 92 92 92 169 169 169 F F F 10 200-219 92 92 92 169 169 169 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 83 83 83 155 155 155 F F F 13 260-279 90 90 90 167 167 167 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 89 89 89 163 163 163 F F F 16 320-339 87 87 87 160 160 160 F F F 17 340-359 92 92 92 169 169 169 F F F 18 360-379 87 87 87 160 160 160 F F F 19 380-399 90 90 90 167 167 167 F F F 20 400-419 88 88 88 162 162 162 F F F 21 420-439 88 88 88 162 162 162 F F F 22 440-459 92 92 92 169 169 169 F F F 23 460-479 92 92 92 169 169 169 F F F 24 480-499 92 92 92 169 169 169 F F F 25 500-519 90 90 90 167 167 167 F F F 26 520-539 86 86 86 158 158 158 F F F 27 540-559 92 92 92 169 169 169 F F F 28 560-579 92 92 92 169 169 169 F F F 29 580-599 82 82 82 153 153 153 F F F Max gen: 92, Max flush: 169 Begin loop 47 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.58031962Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:44.587453017Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.58750444Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.594568627Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.594830079Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.594858975Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.594867533Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.594879132Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513} {"msg":"Crucible a425f5e3-9ddf-4f17-a204-2692b44b0ab2 has session id: bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595188076Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Upstairs opts: Upstairs UUID: a425f5e3-9ddf-4f17-a204-2692b44b0ab2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595220712Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Crucible stats registered with UUID: a425f5e3-9ddf-4f17-a204-2692b44b0ab2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.59523362Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:44.595242718Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595392397Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"io task","client":"1","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595413963Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"io task","client":"0","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595423902Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"io task","client":"2","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595553633Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"task":"crutest"} {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595578799Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513} {"msg":"a425f5e3-9ddf-4f17-a204-2692b44b0ab2 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595591897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595600786Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"client":"0","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595610295Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"client":"1","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595619853Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"client":"2","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595629352Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"io task","client":"2","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595639051Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"io task","client":"0","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595654258Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"io task","client":"1","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"ds_connection connected from Ok(127.0.0.1:33582)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.59571154Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"io task","client":"0","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"ds_connection connected from Ok(127.0.0.1:62585)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595736736Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"io task","client":"2","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"ds_connection connected from Ok(127.0.0.1:48239)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.595750554Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"io task","client":"1","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.5963282Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"client":"1","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.596356526Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"client":"2","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.596691357Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.596716193Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"client":"0","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.596727421Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"[0]R flush_numbers[0..12]: [164, 164, 169, 169, 169, 169, 158, 169, 169, 169, 169, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.596999392Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"[0]R generation[0..12]: [90, 90, 92, 92, 92, 92, 86, 92, 92, 92, 92, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.59701061Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597021928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"[1]R flush_numbers[0..12]: [164, 164, 169, 169, 169, 169, 158, 169, 169, 169, 169, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597032557Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"[1]R generation[0..12]: [90, 90, 92, 92, 92, 92, 86, 92, 92, 92, 92, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597042735Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597053654Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"[2]R flush_numbers[0..12]: [164, 164, 169, 169, 169, 169, 158, 169, 169, 169, 169, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597064142Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"[2]R generation[0..12]: [90, 90, 92, 92, 92, 92, 86, 92, 92, 92, 92, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597074241Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597086979Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Max found gen is 93","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597096178Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Generation requested: 94 >= found:93","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597105686Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Next flush: 170","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597114895Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597124153Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"":"downstairs","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597133832Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597142451Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"client":"0","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597152099Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"client":"1","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597165977Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"client":"2","":"downstairs","session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597175606Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597184665Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"a425f5e3-9ddf-4f17-a204-2692b44b0ab2 is now active with session: bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597194473Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597203312Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597210971Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513,"session_id":"bc1d29b1-2d4f-4e95-b477-d0cbe6e13de1"} {"msg":"Activated sub_volume a425f5e3-9ddf-4f17-a204-2692b44b0ab2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:44.597219979Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 345 len 4096 data: 12 02/30 Read block 322 len 16384 03/30 Read block 73 len 8192 04/30 Read block 20 len 28672 05/30 Write block 313 len 32768 data: 10 10 10 13 13 13 12 11 06/30 Read block 330 len 40960 07/30 Write block 83 len 40960 data: 7 8 10 9 11 10 11 9 9 7 08/30 Read block 295 len 20480 09/30 Flush 10/30 Write block 37 len 24576 data: 6 7 8 9 9 10 11/30 Write block 490 len 8192 data: 5 5 12/30 Read block 274 len 40960 13/30 Read block 274 len 40960 14/30 Read block 318 len 24576 15/30 Flush 16/30 Write block 432 len 16384 data: 6 9 11 11 17/30 Flush 18/30 Write block 405 len 4096 data: 5 19/30 Write block 590 len 40960 data: 9 9 10 9 8 6 6 4 3 2 20/30 Read block 67 len 24576 21/30 Write block 553 len 12288 data: 8 9 10 22/30 Write block 311 len 12288 data: 11 9 11 23/30 Write block 13 len 36864 data: 9 8 7 7 8 7 7 6 6 24/30 Read block 178 len 36864 25/30 Flush 26/30 Write block 192 len 20480 data: 12 12 10 10 9 27/30 Read block 73 len 28672 28/30 Write block 420 len 40960 data: 6 5 8 7 7 6 6 6 5 5 29/30 Write block 549 len 24576 data: 8 7 9 8 9 10 30/30 Write block 242 len 36864 data: 10 10 9 10 10 8 7 6 8 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Read 7 Done Done Done false 1004 Acked Write 8 Done Done Done false 1005 Acked Read 10 Done Done Done false 1006 Acked Write 10 Done Done Done false 1007 Acked Read 5 Done Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 4 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Read 9 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 5 Sent Done Done false 1026 Acked Read 7 Sent Done Done false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 22 3 3 28 Done 8 27 27 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:44.606390309Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1513} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 94 94 164 173 173 F F F <--- 1 020-039 90 94 94 164 173 173 F F F <--- 2 040-059 92 94 94 169 171 171 F F F <--- 3 060-079 92 92 92 169 169 169 F F F 4 080-099 92 94 94 169 170 170 T F F <--- 5 100-119 92 92 92 169 169 169 F F F 6 120-139 86 86 86 158 158 158 F F F 7 140-159 92 92 92 169 169 169 F F F 8 160-179 92 92 92 169 169 169 F F F 9 180-199 92 92 92 169 169 169 F T T <--- 10 200-219 92 92 92 169 169 169 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 83 83 83 155 155 155 F T T <--- 13 260-279 90 90 90 167 167 167 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 89 94 94 163 173 173 T F F <--- 16 320-339 87 94 94 160 170 170 T F F <--- 17 340-359 92 94 94 169 170 170 T F F <--- 18 360-379 87 87 87 160 160 160 F F F 19 380-399 90 90 90 167 167 167 F F F 20 400-419 88 94 94 162 173 173 F F F <--- 21 420-439 88 94 94 162 172 172 F T T <--- 22 440-459 92 92 92 169 169 169 F F F 23 460-479 92 92 92 169 169 169 F F F 24 480-499 92 94 94 169 171 171 F F F <--- 25 500-519 90 90 90 167 167 167 F F F 26 520-539 86 86 86 158 158 158 F F F 27 540-559 92 94 94 169 173 173 F T T <--- 28 560-579 92 92 92 169 169 169 F F F 29 580-599 82 94 94 153 173 173 F F F <--- Max gen: 94, Max flush: 173 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 47 Verifying data now No change after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 173 173 173 F F F 1 020-039 94 94 94 173 173 173 F F F 2 040-059 94 94 94 171 171 171 F F F 3 060-079 92 92 92 169 169 169 F F F 4 080-099 94 94 94 170 170 170 F F F 5 100-119 92 92 92 169 169 169 F F F 6 120-139 86 86 86 158 158 158 F F F 7 140-159 92 92 92 169 169 169 F F F 8 160-179 92 92 92 169 169 169 F F F 9 180-199 95 95 95 174 174 174 F F F 10 200-219 92 92 92 169 169 169 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 95 95 95 174 174 174 F F F 13 260-279 90 90 90 167 167 167 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 94 94 94 173 173 173 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 94 94 94 170 170 170 F F F 18 360-379 87 87 87 160 160 160 F F F 19 380-399 90 90 90 167 167 167 F F F 20 400-419 94 94 94 173 173 173 F F F 21 420-439 95 95 95 174 174 174 F F F 22 440-459 92 92 92 169 169 169 F F F 23 460-479 92 92 92 169 169 169 F F F 24 480-499 94 94 94 171 171 171 F F F 25 500-519 90 90 90 167 167 167 F F F 26 520-539 86 86 86 158 158 158 F F F 27 540-559 95 95 95 174 174 174 F F F 28 560-579 92 92 92 169 169 169 F F F 29 580-599 94 94 94 173 173 173 F F F Max gen: 95, Max flush: 174 Begin loop 48 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.864673869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.873073552Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.873367089Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.873393905Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.873406473Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.873417921Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521} {"msg":"Crucible be1869d8-2e82-4f23-a2c5-2b6a1fbe9876 has session id: 01141eee-d97a-4f46-a935-8ee49de4d138","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.873750013Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Upstairs opts: Upstairs UUID: be1869d8-2e82-4f23-a2c5-2b6a1fbe9876, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.87377051Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Crucible stats registered with UUID: be1869d8-2e82-4f23-a2c5-2b6a1fbe9876","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.873780738Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:48.873789697Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.873949394Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"io task","client":"1","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.873986698Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"io task","client":"0","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.873997567Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"io task","client":"2","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874117509Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"task":"crutest"} {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874134647Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521} {"msg":"be1869d8-2e82-4f23-a2c5-2b6a1fbe9876 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874172981Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874184229Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"client":"0","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874193868Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"client":"1","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874203517Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"client":"2","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874213035Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"io task","client":"2","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.87424881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"io task","client":"1","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874263248Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"io task","client":"0","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"ds_connection connected from Ok(127.0.0.1:43470)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874306712Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"io task","client":"1","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"ds_connection connected from Ok(127.0.0.1:53906)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.87431778Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"io task","client":"2","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"ds_connection connected from Ok(127.0.0.1:40953)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874380341Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"io task","client":"0","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874958086Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"client":"1","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.874986972Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"client":"0","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875088597Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"client":"2","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875253503Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875270531Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"[0]R flush_numbers[0..12]: [173, 173, 171, 169, 170, 169, 158, 169, 169, 174, 169, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875358018Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"[0]R generation[0..12]: [94, 94, 94, 92, 94, 92, 86, 92, 92, 95, 92, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875376475Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875387664Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"[1]R flush_numbers[0..12]: [173, 173, 171, 169, 170, 169, 158, 169, 169, 174, 169, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875398292Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"[1]R generation[0..12]: [94, 94, 94, 92, 94, 92, 86, 92, 92, 95, 92, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.87540866Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875419679Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"[2]R flush_numbers[0..12]: [173, 173, 171, 169, 170, 169, 158, 169, 169, 174, 169, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875430257Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"[2]R generation[0..12]: [94, 94, 94, 92, 94, 92, 86, 92, 92, 95, 92, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875440376Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875451144Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875460393Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875469962Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Next flush: 175","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.87547927Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875488479Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"":"downstairs","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875498137Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875506776Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"client":"0","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875517585Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"client":"1","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible"Wait for a query_work_queue command to finish before sending IO ,"level":30,"time":"2026-04-15T18:13:48.875531323Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"client":"2","":"downstairs","session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875541331Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.87555046Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"be1869d8-2e82-4f23-a2c5-2b6a1fbe9876 is now active with session: 01141eee-d97a-4f46-a935-8ee49de4d138","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875560168Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875569017Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875576486Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521,"session_id":"01141eee-d97a-4f46-a935-8ee49de4d138"} {"msg":"Activated sub_volume be1869d8-2e82-4f23-a2c5-2b6a1fbe9876","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:48.875585445Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 70 len 12288 data: 4 5 5 02/30 Read block 217 len 32768 03/30 Read block 281 len 28672 04/30 Read block 202 len 8192 05/30 Write block 136 len 32768 data: 4 4 6 10 14 15 13 15 06/30 Read block 250 len 32768 07/30 Read block 110 len 20480 08/30 Flush 09/30 Write block 425 len 28672 data: 7 7 7 6 6 6 8 10/30 Read block 369 len 28672 11/30 Read block 551 len 40960 12/30 Read block 488 len 36864 13/30 Write block 466 len 24576 data: 7 9 11 10 10 9 14/30 Read block 254 len 32768 15/30 Read block 260 len 28672 16/30 Read block 514 len 12288 17/30 Read block 122 len 8192 18/30 Read block 136 len 36864 19/30 Read block 363 len 24576 20/30 Write block 62 len 32768 data: 6 6 8 8 6 5 5 5 21/30 Flush 22/30 Write block 353 len 20480 data: 10 10 10 10 10 23/30 Read block 322 len 28672 24/30 Read block 27 len 24576 25/30 Read block 555 len 20480 26/30 Read block 303 len 24576 27/30 Read block 142 len 36864 28/30 Read block 228 len 24576 29/30 Read block 72 len 20480 30/30 Read block 258 len 40960 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 8 Done Done Sent false 1002 Acked Read 7 Done Done Sent false 1003 Acked Read 2 Done Done Sent false 1004 Acked Write 8 Done Done Sent false 1005 Acked Read 8 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Read 10 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Read 6 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 6 Done Done Sent false 1024 Acked Read 5 Done Done Sent false 1025 Acked Read 6 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:48.885782394Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1521} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 173 173 173 F F F 1 020-039 94 94 94 173 173 173 F F F 2 040-059 94 94 94 171 171 171 F F F 3 060-079 96 96 92 176 176 169 F F T <--- 4 080-099 94 94 94 170 170 170 F F F 5 100-119 92 92 92 169 169 169 F F F 6 120-139 96 96 86 175 175 158 F F F <--- 7 140-159 96 96 92 175 175 169 F F F <--- 8 160-179 92 92 92 169 169 169 F F F 9 180-199 95 95 95 174 174 174 F F F 10 200-219 92 92 92 169 169 169 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 95 95 95 174 174 174 F F F 13 260-279 90 90 90 167 167 167 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 94 94 94 173 173 173 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 94 94 94 170 170 170 T T F <--- 18 360-379 87 87 87 160 160 160 F F F 19 380-399 90 90 90 167 167 167 F F F 20 400-419 94 94 94 173 173 173 F F F 21 420-439 96 96 95 176 176 174 F F F <--- 22 440-459 92 92 92 169 169 169 F F F 23 460-479 96 96 92 176 176 169 F F F <--- 24 480-499 94 94 94 171 171 171 F F F 25 500-519 90 90 90 167 167 167 F F F 26 520-539 86 86 86 158 158 158 F F F 27 540-559 95 95 95 174 174 174 F F F 28 560-579 92 92 92 169 169 169 F F F 29 580-599 94 94 94 173 173 173 F F F Max gen: 96, Max flush: 176 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 173 173 173 F F F 1 020-039 94 94 94 173 173 173 F F F 2 040-059 94 94 94 171 171 171 F F F 3 060-079 96 96 96 176 176 176 F F F 4 080-099 94 94 94 170 170 170 F F F 5 100-119 92 92 92 169 169 169 F F F 6 120-139 96 96 96 175 175 175 F F F 7 140-159 96 96 96 175 175 175 F F F 8 160-179 92 92 92 169 169 169 F F F 9 180-199 95 95 95 174 174 174 F F F 10 200-219 92 92 92 169 169 169 F F F 11 220-239 85 85 85 157 157 157 F F F 12 240-259 95 95 95 174 174 174 F F F 13 260-279 90 90 90 167 167 167 F F F 14 280-299 84 84 84 156 156 156 F F F 15 300-319 94 94 94 173 173 173 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 97 97 97 177 177 177 F F F 18 360-379 87 87 87 160 160 160 F F F 19 380-399 90 90 90 167 167 167 F F F 20 400-419 94 94 94 173 173 173 F F F 21 420-439 96 96 96 176 176 176 F F F 22 440-459 92 92 92 169 169 169 F F F 23 460-479 96 96 96 176 176 176 F F F 24 480-499 94 94 94 171 171 171 F F F 25 500-519 90 90 90 167 167 167 F F F 26 520-539 86 86 86 158 158 158 F F F 27 540-559 95 95 95 174 174 174 F F F 28 560-579 92 92 92 169 169 169 F F F 29 580-599 94 94 94 173 173 173 F F F Max gen: 97, Max flush: 177 Begin loop 49 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.121990402Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:53.129219335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.129282176Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.136132575Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.136396176Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.136422532Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.136430931Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.136442469Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529} {"msg":"Crucible b09783b5-74c2-4e1b-ab1e-77889f044ccc has session id: 847badb9-0809-473d-8103-0793a6aa88e9","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.13678311Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Upstairs opts: Upstairs UUID: b09783b5-74c2-4e1b-ab1e-77889f044ccc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.136813035Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Crucible stats registered with UUID: b09783b5-74c2-4e1b-ab1e-77889f044ccc","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.136824654Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:53.136833872Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.1369883Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"io task","client":"1","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137014146Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"io task","client":"0","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137024254Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"io task","client":"2","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137155055Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"task":"crutest"} {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137172693Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529} {"msg":"b09783b5-74c2-4e1b-ab1e-77889f044ccc active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137210537Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137219756Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"client":"0","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137229344Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"client":"1","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137238773Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"client":"2","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137248202Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"io task","client":"2","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.13725817Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"io task","client":"1","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137267999Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"io task","client":"0","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:50475)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137345787Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"io task","client":"0","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:53428)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137371324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"io task","client":"2","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:56481)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.137382892Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"io task","client":"1","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138054254Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"client":"2","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138075811Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"client":"0","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138163418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"client":"1","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.13835484Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138364738Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"[0]R flush_numbers[0..12]: [173, 173, 171, 176, 170, 169, 175, 175, 169, 174, 169, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138440957Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"[0]R generation[0..12]: [94, 94, 94, 96, 94, 92, 96, 96, 92, 95, 92, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138452376Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138463354Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"[1]R flush_numbers[0..12]: [173, 173, 171, 176, 170, 169, 175, 175, 169, 174, 169, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138473952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"[1]R generation[0..12]: [94, 94, 94, 96, 94, 92, 96, 96, 92, 95, 92, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138484131Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138495109Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"[2]R flush_numbers[0..12]: [173, 173, 171, 176, 170, 169, 175, 175, 169, 174, 169, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138507818Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"[2]R generation[0..12]: [94, 94, 94, 96, 94, 92, 96, 96, 92, 95, 92, 85]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138518126Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138528864Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138538173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138547762Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Next flush: 178","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.13855694Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138566199Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"":"downstairs","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138575708Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138584386Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"client":"0","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138593995Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"client":"1","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138603454Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"client":"2","":"downstairs","session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138613072Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.13862646Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"b09783b5-74c2-4e1b-ab1e-77889f044ccc is now active with session: 847badb9-0809-473d-8103-0793a6aa88e9","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138636479Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138645537Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138653376Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529,"session_id":"847badb9-0809-473d-8103-0793a6aa88e9"} {"msg":"Activated sub_volume b09783b5-74c2-4e1b-ab1e-77889f044ccc","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:53.138664375Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 14 len 8192 data: 9 8 02/30 Write block 457 len 8192 data: 12 14 03/30 Read block 568 len 28672 04/30 Flush 05/30 Write block 294 len 4096 data: 7 06/30 Read block 380 len 32768 07/30 Read block 419 len 8192 08/30 Read block 554 len 24576 09/30 Read block 265 len 36864 10/30 Write block 553 len 36864 data: 10 11 11 11 12 11 9 10 10 11/30 Write block 373 len 40960 data: 9 9 7 10 10 7 8 7 6 7 12/30 Read block 83 len 12288 13/30 Read block 553 len 20480 14/30 Read block 172 len 8192 15/30 Write block 533 len 36864 data: 14 13 14 12 12 11 10 8 8 16/30 Write block 131 len 28672 data: 6 8 8 6 5 5 5 17/30 Write block 414 len 4096 data: 6 18/30 Read block 246 len 24576 19/30 Flush 20/30 Write block 305 len 20480 data: 7 10 6 6 9 21/30 Write block 356 len 12288 data: 11 11 10 22/30 Read block 408 len 4096 23/30 Write block 159 len 36864 data: 8 7 7 8 15 14 13 11 11 24/30 Read block 536 len 32768 25/30 Flush 26/30 Write block 222 len 40960 data: 7 10 10 9 11 11 12 12 11 7 27/30 Read block 487 len 4096 28/30 Write block 391 len 12288 data: 12 11 11 29/30 Write block 183 len 4096 data: 11 30/30 Write block 432 len 40960 data: 7 10 12 12 9 9 12 11 10 12 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 1 Done Done Done false 1005 Acked Read 8 Done Done Done false 1006 Acked Read 2 Done Done Done false 1007 Acked Read 6 Done Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Write 9 Sent Done Done false 1010 Acked Write 10 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Read 1 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 3 Sent Sent Done false 1028 Acked Write 1 Sent Sent Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 22 3 1 26 Done 4 23 25 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:53.147746777Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1529} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 98 98 178 178 178 F F F 1 020-039 94 94 94 173 173 173 F F F 2 040-059 94 94 94 171 171 171 F F F 3 060-079 96 96 96 176 176 176 F F F 4 080-099 94 94 94 170 170 170 F F F 5 100-119 92 92 92 169 169 169 F F F 6 120-139 96 98 98 175 179 179 F F F <--- 7 140-159 96 98 98 175 180 180 F F F <--- 8 160-179 92 98 98 169 180 180 F F F <--- 9 180-199 95 95 95 174 174 174 F T T <--- 10 200-219 92 92 92 169 169 169 F F F 11 220-239 85 85 85 157 157 157 F T T <--- 12 240-259 95 95 95 174 174 174 F F F 13 260-279 90 90 90 167 167 167 F F F 14 280-299 84 98 98 156 179 179 T F F <--- 15 300-319 94 98 98 173 180 180 F F F <--- 16 320-339 94 94 94 170 170 170 F F F 17 340-359 97 98 98 177 180 180 F F F <--- 18 360-379 87 98 98 160 179 179 F F F <--- 19 380-399 90 98 98 167 179 179 F T T <--- 20 400-419 94 98 98 173 179 179 F F F <--- 21 420-439 96 96 96 176 176 176 F T T <--- 22 440-459 98 98 98 178 178 178 F T T <--- 23 460-479 96 96 96 176 176 176 F F F 24 480-499 94 94 94 171 171 171 F F F 25 500-519 90 90 90 167 167 167 F F F 26 520-539 86 98 98 158 179 179 F F F <--- 27 540-559 95 98 98 174 179 179 F F F <--- 28 560-579 92 98 98 169 179 179 F F F <--- 29 580-599 94 94 94 173 173 173 F F F Max gen: 98, Max flush: 180 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 98 98 178 178 178 F F F 1 020-039 94 94 94 173 173 173 F F F 2 040-059 94 94 94 171 171 171 F F F 3 060-079 96 96 96 176 176 176 F F F 4 080-099 94 94 94 170 170 170 F F F 5 100-119 92 92 92 169 169 169 F F F 6 120-139 98 98 98 179 179 179 F F F 7 140-159 98 98 98 180 180 180 F F F 8 160-179 98 98 98 180 180 180 F F F 9 180-199 99 99 99 181 181 181 F F F 10 200-219 92 92 92 169 169 169 F F F 11 220-239 99 99 99 181 181 181 F F F 12 240-259 95 95 95 174 174 174 F F F 13 260-279 90 90 90 167 167 167 F F F 14 280-299 98 98 98 179 179 179 F F F 15 300-319 98 98 98 180 180 180 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 98 98 98 180 180 180 F F F 18 360-379 98 98 98 179 179 179 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 98 98 98 179 179 179 F F F 21 420-439 99 99 99 181 181 181 F F F 22 440-459 99 99 99 181 181 181 F F F 23 460-479 96 96 96 176 176 176 F F F 24 480-499 94 94 94 171 171 171 F F F 25 500-519 90 90 90 167 167 167 F F F 26 520-539 98 98 98 179 179 179 F F F 27 540-559 98 98 98 179 179 179 F F F 28 560-579 98 98 98 179 179 179 F F F 29 580-599 94 94 94 173 173 173 F F F Max gen: 99, Max flush: 181 Begin loop 50 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.410621227Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:57.417820525Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.417882416Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.42469747Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.424963751Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.424990217Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.424999006Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425010384Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537} {"msg":"Crucible ed7feafa-cdcd-4526-9936-423acb8f675b has session id: f763b44d-7f4f-4677-8cfc-5f2c96b581ab","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425353224Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Upstairs opts: Upstairs UUID: ed7feafa-cdcd-4526-9936-423acb8f675b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.42538188Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Crucible stats registered with UUID: ed7feafa-cdcd-4526-9936-423acb8f675b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425393318Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:57.425402477Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425565033Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"io task","client":"2","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425601147Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"io task","client":"0","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425613946Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"io task","client":"1","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425683795Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"task":"crutest"} {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425696874Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537} {"msg":"ed7feafa-cdcd-4526-9936-423acb8f675b active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.42572058Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425730039Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"client":"0","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425739707Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"client":"1","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425749176Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"client":"2","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425758714Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"io task","client":"2","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425772532Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"io task","client":"1","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:62030)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425821595Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"io task","client":"2","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425832054Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"io task","client":"0","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:48848)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425842192Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"io task","client":"1","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:41540)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.425896704Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"io task","client":"0","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.42654237Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"client":"1","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.426605661Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"client":"2","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.426618349Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"client":"0","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.426913626Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.42695058Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"[0]R flush_numbers[0..12]: [178, 173, 171, 176, 170, 169, 179, 180, 180, 181, 169, 181]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.426985415Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"[0]R generation[0..12]: [98, 94, 94, 96, 94, 92, 98, 98, 98, 99, 92, 99]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.426996454Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427007592Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"[1]R flush_numbers[0..12]: [178, 173, 171, 176, 170, 169, 179, 180, 180, 181, 169, 181]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.4270183Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"[1]R generation[0..12]: [98, 94, 94, 96, 94, 92, 98, 98, 98, 99, 92, 99]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427032458Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427043247Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"[2]R flush_numbers[0..12]: [178, 173, 171, 176, 170, 169, 179, 180, 180, 181, 169, 181]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427053675Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"[2]R generation[0..12]: [98, 94, 94, 96, 94, 92, 98, 98, 98, 99, 92, 99]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427063944Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427074732Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427084171Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427093769Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Next flush: 182","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427102968Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427113796Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"":"downstairs","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427123725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427132454Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"client":"0","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427142322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"client":"1","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427152141Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"client":"2","":"downstairs","session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"All required reconciliation work is completed","v":0,"Wait for a query_work_queue command to finish before sending IO name":"crucible","level":30,"time":"2026-04-15T18:13:57.427165689Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427174817Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"ed7feafa-cdcd-4526-9936-423acb8f675b is now active with session: f763b44d-7f4f-4677-8cfc-5f2c96b581ab","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427184696Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427193815Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427201424Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537,"session_id":"f763b44d-7f4f-4677-8cfc-5f2c96b581ab"} {"msg":"Activated sub_volume ed7feafa-cdcd-4526-9936-423acb8f675b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:13:57.427210322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 47 len 20480 data: 12 11 11 8 7 02/30 Write block 157 len 16384 data: 6 7 9 8 03/30 Read block 66 len 40960 04/30 Write block 514 len 8192 data: 9 7 05/30 Write block 471 len 16384 data: 10 9 10 10 06/30 Read block 58 len 12288 07/30 Write block 11 len 20480 data: 9 9 10 10 9 08/30 Read block 191 len 12288 09/30 Read block 32 len 32768 10/30 Write block 468 len 40960 data: 12 11 11 11 10 11 11 8 6 4 11/30 Read block 224 len 20480 12/30 Read block 399 len 28672 13/30 Write block 487 len 4096 data: 3 14/30 Read block 554 len 12288 15/30 Read block 273 len 28672 16/30 Read block 163 len 36864 17/30 Read block 469 len 12288 18/30 Read block 286 len 36864 19/30 Write block 266 len 36864 data: 14 14 12 12 9 10 10 10 8 20/30 Flush 21/30 Write block 25 len 8192 data: 8 7 22/30 Write block 480 len 40960 data: 5 7 8 8 6 5 4 4 3 4 23/30 Read block 433 len 40960 24/30 Write block 520 len 32768 data: 7 8 8 7 7 7 9 11 25/30 Write block 0 len 40960 data: 3 4 6 6 8 8 8 8 8 8 26/30 Read block 240 len 20480 27/30 Read block 255 len 20480 28/30 Read block 396 len 40960 29/30 Read block 472 len 4096 30/30 Read block 310 len 32768 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Sent Done Done false 1001 Acked Write 4 Sent Done Done false 1002 Acked Read 10 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Read 7 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Read 10 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Read 8 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 0 31 Done 0 29 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:13:57.437224429Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1537} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 100 100 178 182 182 F T T <--- 1 020-039 94 94 94 173 173 173 F T T <--- 2 040-059 94 100 100 171 182 182 F F F <--- 3 060-079 96 96 96 176 176 176 F F F 4 080-099 94 94 94 170 170 170 F F F 5 100-119 92 92 92 169 169 169 F F F 6 120-139 98 98 98 179 179 179 F F F 7 140-159 98 100 100 180 182 182 F F F <--- 8 160-179 98 100 100 180 182 182 F F F <--- 9 180-199 99 99 99 181 181 181 F F F 10 200-219 92 92 92 169 169 169 F F F 11 220-239 99 99 99 181 181 181 F F F 12 240-259 95 95 95 174 174 174 F F F 13 260-279 90 100 100 167 182 182 F F F <--- 14 280-299 98 98 98 179 179 179 F F F 15 300-319 98 98 98 180 180 180 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 98 98 98 180 180 180 F F F 18 360-379 98 98 98 179 179 179 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 98 98 98 179 179 179 F F F 21 420-439 99 99 99 181 181 181 F F F 22 440-459 99 99 99 181 181 181 F F F 23 460-479 96 100 100 176 182 182 F F F <--- 24 480-499 94 100 100 171 182 182 F T T <--- 25 500-519 90 100 100 167 182 182 F F F <--- 26 520-539 98 98 98 179 179 179 F T T <--- 27 540-559 98 98 98 179 179 179 F F F 28 560-579 98 98 98 179 179 179 F F F 29 580-599 94 94 94 173 173 173 F F F Max gen: 100, Max flush: 182 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 101 101 101 183 183 183 F F F 1 020-039 101 101 101 183 183 183 F F F 2 040-059 100 100 100 182 182 182 F F F 3 060-079 96 96 96 176 176 176 F F F 4 080-099 94 94 94 170 170 170 F F F 5 100-119 92 92 92 169 169 169 F F F 6 120-139 98 98 98 179 179 179 F F F 7 140-159 100 100 100 182 182 182 F F F 8 160-179 100 100 100 182 182 182 F F F 9 180-199 99 99 99 181 181 181 F F F 10 200-219 92 92 92 169 169 169 F F F 11 220-239 99 99 99 181 181 181 F F F 12 240-259 95 95 95 174 174 174 F F F 13 260-279 100 100 100 182 182 182 F F F 14 280-299 98 98 98 179 179 179 F F F 15 300-319 98 98 98 180 180 180 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 98 98 98 180 180 180 F F F 18 360-379 98 98 98 179 179 179 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 98 98 98 179 179 179 F F F 21 420-439 99 99 99 181 181 181 F F F 22 440-459 99 99 99 181 181 181 F F F 23 460-479 100 100 100 182 182 182 F F F 24 480-499 101 101 101 183 183 183 F F F 25 500-519 100 100 100 182 182 182 F F F 26 520-539 101 101 101 183 183 183 F F F 27 540-559 98 98 98 179 179 179 F F F 28 560-579 98 98 98 179 179 179 F F F 29 580-599 94 94 94 173 173 173 F F F Max gen: 101, Max flush: 183 Begin loop 51 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.689516215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.697739773Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.69803443Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698061286Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698072475Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698083903Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545} {"msg":"Crucible 1b0e2a99-c2ef-483e-bb97-2d3a6bcd9973 has session id: 990bf26f-ef72-426c-a761-5d74d68717a7","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.69837668Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Upstairs opts: Upstairs UUID: 1b0e2a99-c2ef-483e-bb97-2d3a6bcd9973, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698401037Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Crucible stats registered with UUID: 1b0e2a99-c2ef-483e-bb97-2d3a6bcd9973","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698411075Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:01.698420074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.69858639Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"io task","client":"0","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698624304Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"io task","client":"1","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698635562Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"io task","client":"2","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698807557Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"task":"crutest"} {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698821365Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545} {"msg":"1b0e2a99-c2ef-483e-bb97-2d3a6bcd9973 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.69885409Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698863469Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"client":"0","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698873168Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"client":"1","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698882836Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"client":"2","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698892405Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"io task","client":"2","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698909322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"io task","client":"1","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698943217Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"io task","client":"0","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"ds_connection connected from Ok(127.0.0.1:44993)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698954046Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"io task","client":"2","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"ds_connection connected from Ok(127.0.0.1:53675)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.698968294Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"io task","client":"1","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"ds_connection connected from Ok(127.0.0.1:62511)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.699045702Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"io task","client":"0","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.699578125Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"client":"0","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.699601651Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"client":"1","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.69967572Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"client":"2","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.699910896Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.699920435Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"[0]R flush_numbers[0..12]: [183, 183, 182, 176, 170, 169, 179, 182, 182, 181, 169, 181]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700003162Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"[0]R generation[0..12]: [101, 101, 100, 96, 94, 92, 98, 100, 100, 99, 92, 99]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700014381Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700025359Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"[1]R flush_numbers[0..12]: [183, 183, 182, 176, 170, 169, 179, 182, 182, 181, 169, 181]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700035928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"[1]R generation[0..12]: [101, 101, 100, 96, 94, 92, 98, 100, 100, 99, 92, 99]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700048516Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700059454Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"[2]R flush_numbers[0..12]: [183, 183, 182, 176, 170, 169, 179, 182, 182, 181, 169, 181]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700069933Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"[2]R generation[0..12]: [101, 101, 100, 96, 94, 92, 98, 100, 100, 99, 92, 99]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700080071Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.70009096Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700100328Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700110037Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Next flush: 184","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700119376Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700128614Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"":"downstairs","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700138123Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700146752Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"client":"0","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.70015631Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"client":"1","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Wait for a query_work_queue command to finish before sending IO Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700170058Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"client":"2","":"downstairs","session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700179847Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700188815Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"1b0e2a99-c2ef-483e-bb97-2d3a6bcd9973 is now active with session: 990bf26f-ef72-426c-a761-5d74d68717a7","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700199904Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700208762Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.700216211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545,"session_id":"990bf26f-ef72-426c-a761-5d74d68717a7"} {"msg":"Activated sub_volume 1b0e2a99-c2ef-483e-bb97-2d3a6bcd9973","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:01.70022523Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 259 len 12288 data: 6 4 7 02/30 Write block 523 len 40960 data: 8 8 8 10 12 11 11 10 11 13 03/30 Read block 159 len 20480 04/30 Read block 165 len 40960 05/30 Read block 451 len 4096 06/30 Write block 556 len 4096 data: 12 07/30 Read block 87 len 8192 08/30 Write block 565 len 12288 data: 9 7 8 09/30 Write block 574 len 40960 data: 8 10 11 8 7 6 5 5 4 6 10/30 Write block 456 len 8192 data: 12 13 11/30 Write block 111 len 16384 data: 10 7 6 8 12/30 Write block 164 len 16384 data: 15 14 12 12 13/30 Read block 569 len 36864 14/30 Write block 578 len 28672 data: 8 7 6 6 5 7 9 15/30 Write block 125 len 36864 data: 6 7 5 4 5 6 7 9 9 16/30 Write block 61 len 20480 data: 7 7 7 9 9 17/30 Write block 430 len 16384 data: 7 9 8 11 18/30 Flush 19/30 Write block 579 len 8192 data: 8 7 20/30 Read block 58 len 36864 21/30 Write block 134 len 36864 data: 7 6 6 6 7 11 15 16 14 22/30 Flush 23/30 Write block 293 len 16384 data: 10 8 5 5 24/30 Read block 400 len 20480 25/30 Read block 134 len 16384 26/30 Write block 197 len 32768 data: 10 9 9 8 7 5 5 9 27/30 Read block 487 len 24576 28/30 Read block 527 len 16384 29/30 Write block 117 len 32768 data: 11 9 8 10 9 10 8 7 30/30 Write block 308 len 24576 data: 7 10 9 12 10 12 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 9 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 4 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 12 16 Done 10 10 0 20 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) Some(JobId(1017)) Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:01.70877556Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1545} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 101 101 101 183 183 183 F F F 1 020-039 101 101 101 183 183 183 F F F 2 040-059 100 100 100 182 182 182 F F F 3 060-079 102 102 102 184 184 184 F F F 4 080-099 94 94 94 170 170 170 F F F 5 100-119 102 102 102 184 184 184 T T F <--- 6 120-139 102 102 102 185 185 184 T T F <--- 7 140-159 102 102 100 185 185 182 F F F <--- 8 160-179 102 102 102 184 184 184 F F F 9 180-199 99 99 99 181 181 181 T T F <--- 10 200-219 92 92 92 169 169 169 T T F <--- 11 220-239 99 99 99 181 181 181 F F F 12 240-259 102 102 102 184 184 184 F F F 13 260-279 102 102 102 184 184 184 F F F 14 280-299 98 98 98 179 179 179 T T F <--- 15 300-319 98 98 98 180 180 180 T T F <--- 16 320-339 94 94 94 170 170 170 F F F 17 340-359 98 98 98 180 180 180 F F F 18 360-379 98 98 98 179 179 179 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 98 98 98 179 179 179 F F F 21 420-439 102 102 102 184 184 184 F F F 22 440-459 102 102 102 184 184 184 F F F 23 460-479 100 100 100 182 182 182 F F F 24 480-499 101 101 101 183 183 183 F F F 25 500-519 100 100 100 182 182 182 F F F 26 520-539 102 102 102 184 184 184 F F F 27 540-559 102 102 102 184 184 184 F F F 28 560-579 102 102 102 185 185 184 F F F <--- 29 580-599 102 102 102 185 185 184 F F F <--- Max gen: 102, Max flush: 185 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 101 101 101 183 183 183 F F F 1 020-039 101 101 101 183 183 183 F F F 2 040-059 100 100 100 182 182 182 F F F 3 060-079 102 102 102 184 184 184 F F F 4 080-099 94 94 94 170 170 170 F F F 5 100-119 103 103 103 186 186 186 F F F 6 120-139 103 103 103 186 186 186 F F F 7 140-159 102 102 102 185 185 185 F F F 8 160-179 102 102 102 184 184 184 F F F 9 180-199 103 103 103 186 186 186 F F F 10 200-219 103 103 103 186 186 186 F F F 11 220-239 99 99 99 181 181 181 F F F 12 240-259 102 102 102 184 184 184 F F F 13 260-279 102 102 102 184 184 184 F F F 14 280-299 103 103 103 186 186 186 F F F 15 300-319 103 103 103 186 186 186 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 98 98 98 180 180 180 F F F 18 360-379 98 98 98 179 179 179 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 98 98 98 179 179 179 F F F 21 420-439 102 102 102 184 184 184 F F F 22 440-459 102 102 102 184 184 184 F F F 23 460-479 100 100 100 182 182 182 F F F 24 480-499 101 101 101 183 183 183 F F F 25 500-519 100 100 100 182 182 182 F F F 26 520-539 102 102 102 184 184 184 F F F 27 540-559 102 102 102 184 184 184 F F F 28 560-579 102 102 102 185 185 185 F F F 29 580-599 102 102 102 185 185 185 F F F Max gen: 103, Max flush: 186 Begin loop 52 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.957470792Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:05.964643564Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.964698926Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.971571171Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.971847701Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.971875027Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.971883686Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.971894924Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553} {"msg":"Crucible 48ec6f4c-ee9d-41d3-807d-b70fd66f12f7 has session id: e60a204f-9582-4413-9040-231412143fa4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.97219322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Upstairs opts: Upstairs UUID: 48ec6f4c-ee9d-41d3-807d-b70fd66f12f7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972217497Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Crucible stats registered with UUID: 48ec6f4c-ee9d-41d3-807d-b70fd66f12f7","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972227555Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:05.972236744Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972405319Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"io task","client":"2","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972428676Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"io task","client":"0","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972440924Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"io task","client":"1","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972527151Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"task":"crutest"} {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972544259Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553} {"msg":"48ec6f4c-ee9d-41d3-807d-b70fd66f12f7 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972577644Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972586923Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"client":"0","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972596501Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"client":"1","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.97260596Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"client":"2","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972615369Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"io task","client":"2","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972630106Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"io task","client":"1","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"ds_connection connected from Ok(127.0.0.1:43965)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.97267127Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"io task","client":"2","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972681869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"io task","client":"0","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"ds_connection connected from Ok(127.0.0.1:45996)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.972692097Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"io task","client":"1","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"ds_connection connected from Ok(127.0.0.1:41968)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.97273909Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"io task","client":"0","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973447277Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"client":"1","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973480312Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"client":"0","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.97349192Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"client":"2","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973798776Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973808854Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"[0]R flush_numbers[0..12]: [183, 183, 182, 184, 170, 186, 186, 185, 184, 186, 186, 181]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.97383504Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"[0]R generation[0..12]: [101, 101, 100, 102, 94, 103, 103, 102, 102, 103, 103, 99]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973846379Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973857097Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"[1]R flush_numbers[0..12]: [183, 183, 182, 184, 170, 186, 186, 185, 184, 186, 186, 181]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973867596Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"[1]R generation[0..12]: [101, 101, 100, 102, 94, 103, 103, 102, 102, 103, 103, 99]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973878024Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973888852Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"[2]R flush_numbers[0..12]: [183, 183, 182, 184, 170, 186, 186, 185, 184, 186, 186, 181]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973899461Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"[2]R generation[0..12]: [101, 101, 100, 102, 94, 103, 103, 102, 102, 103, 103, 99]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973909819Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973920668Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Max found gen is 104","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973930026Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Generation requested: 104 >= found:104","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973939815Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Next flush: 187","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973949134Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973959702Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"":"downstairs","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973969321Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.973977999Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"client":"0","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-04-15T18:14:05.973987708Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"client":"1","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.974000976Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"client":"2","":"downstairs","session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.974010735Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.974019773Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"48ec6f4c-ee9d-41d3-807d-b70fd66f12f7 is now active with session: e60a204f-9582-4413-9040-231412143fa4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.974029542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.974038451Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.974045959Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553,"session_id":"e60a204f-9582-4413-9040-231412143fa4"} {"msg":"Activated sub_volume 48ec6f4c-ee9d-41d3-807d-b70fd66f12f7","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:05.974055108Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 225 len 36864 data: 10 12 12 13 13 12 8 5 4 02/30 Flush 03/30 Write block 537 len 4096 data: 13 04/30 Read block 341 len 28672 05/30 Read block 154 len 24576 06/30 Flush 07/30 Flush 08/30 Write block 41 len 16384 data: 10 11 11 10 09/30 Flush 10/30 Write block 345 len 12288 data: 13 12 10 11/30 Write block 18 len 16384 data: 8 8 7 7 12/30 Read block 151 len 28672 13/30 Read block 424 len 12288 14/30 Read block 544 len 4096 15/30 Write block 137 len 16384 data: 7 8 12 16 16/30 Write block 47 len 40960 data: 13 12 12 9 8 8 7 8 8 10 17/30 Flush 18/30 Flush 19/30 Write block 405 len 36864 data: 6 7 6 6 5 7 7 6 6 20/30 Read block 86 len 12288 21/30 Read block 269 len 4096 22/30 Read block 492 len 32768 23/30 Read block 563 len 32768 24/30 Read block 90 len 20480 25/30 Flush 26/30 Write block 81 len 32768 data: 8 9 8 9 11 10 12 11 27/30 Write block 225 len 40960 data: 11 13 13 14 14 13 9 6 5 4 28/30 Read block 342 len 12288 29/30 Write block 576 len 28672 data: 12 9 9 9 8 7 6 30/30 Read block 231 len 20480 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 1 Sent Done Done false 1003 Acked Read 7 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 4 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Write 7 Sent Done Done false 1029 Acked Read 5 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:05.983422119Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1553} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 101 104 104 183 191 191 F F F <--- 1 020-039 101 104 104 183 191 191 F F F <--- 2 040-059 100 104 104 182 191 191 F F F <--- 3 060-079 102 102 102 184 184 184 F F F 4 080-099 94 94 94 170 170 170 F T T <--- 5 100-119 103 103 103 186 186 186 F F F 6 120-139 103 104 104 186 191 191 F F F <--- 7 140-159 102 104 104 185 191 191 F F F <--- 8 160-179 102 102 102 184 184 184 F F F 9 180-199 103 103 103 186 186 186 F F F 10 200-219 103 103 103 186 186 186 F F F 11 220-239 99 104 104 181 187 187 T T T <--- 12 240-259 102 102 102 184 184 184 F F F 13 260-279 102 102 102 184 184 184 F F F 14 280-299 103 103 103 186 186 186 F F F 15 300-319 103 103 103 186 186 186 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 98 104 104 180 191 191 F F F <--- 18 360-379 98 98 98 179 179 179 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 98 104 104 179 193 193 F F F <--- 21 420-439 102 102 102 184 184 184 F F F 22 440-459 102 102 102 184 184 184 F F F 23 460-479 100 100 100 182 182 182 F F F 24 480-499 101 101 101 183 183 183 F F F 25 500-519 100 100 100 182 182 182 F F F 26 520-539 102 104 104 184 188 188 F F F <--- 27 540-559 102 102 102 184 184 184 F F F 28 560-579 102 102 102 185 185 185 F T T <--- 29 580-599 102 102 102 185 185 185 F T T <--- Max gen: 104, Max flush: 193 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 191 191 191 F F F 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 102 102 102 184 184 184 F F F 4 080-099 105 105 105 194 194 194 F F F 5 100-119 103 103 103 186 186 186 F F F 6 120-139 104 104 104 191 191 191 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 102 102 102 184 184 184 F F F 9 180-199 103 103 103 186 186 186 F F F 10 200-219 103 103 103 186 186 186 F F F 11 220-239 105 105 105 194 194 194 F F F 12 240-259 102 102 102 184 184 184 F F F 13 260-279 102 102 102 184 184 184 F F F 14 280-299 103 103 103 186 186 186 F F F 15 300-319 103 103 103 186 186 186 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 104 104 104 191 191 191 F F F 18 360-379 98 98 98 179 179 179 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 104 104 104 193 193 193 F F F 21 420-439 102 102 102 184 184 184 F F F 22 440-459 102 102 102 184 184 184 F F F 23 460-479 100 100 100 182 182 182 F F F 24 480-499 101 101 101 183 183 183 F F F 25 500-519 100 100 100 182 182 182 F F F 26 520-539 104 104 104 188 188 188 F F F 27 540-559 102 102 102 184 184 184 F F F 28 560-579 105 105 105 194 194 194 F F F 29 580-599 105 105 105 194 194 194 F F F Max gen: 105, Max flush: 194 Begin loop 53 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.25498098Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.263276537Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.263563935Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.263591461Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.26360103Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.263616587Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561} {"msg":"Crucible ce2921e2-8dfc-42ca-a54d-f67eab5206ec has session id: e7741e41-7e62-4cc3-a420-e8485ea83c0f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.263962917Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Upstairs opts: Upstairs UUID: ce2921e2-8dfc-42ca-a54d-f67eab5206ec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.263984644Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Crucible stats registered with UUID: ce2921e2-8dfc-42ca-a54d-f67eab5206ec","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.263995572Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:10.264005841Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264170456Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"io task","client":"0","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264207251Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"io task","client":"2","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264218019Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"io task","client":"1","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264335742Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"task":"crutest"} {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264361539Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561} {"msg":"ce2921e2-8dfc-42ca-a54d-f67eab5206ec active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264398993Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264408322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"client":"0","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.26441785Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"client":"1","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264427589Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"client":"2","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264444876Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"io task","client":"2","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264457025Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"io task","client":"0","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264471782Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"io task","client":"1","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"ds_connection connected from Ok(127.0.0.1:57328)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264511027Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"io task","client":"2","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"ds_connection connected from Ok(127.0.0.1:55222)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264547041Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"io task","client":"0","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"ds_connection connected from Ok(127.0.0.1:57478)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.264686121Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"io task","client":"1","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265268206Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"client":"2","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265281914Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"client":"0","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265292812Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"client":"1","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265574361Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265614845Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"[0]R flush_numbers[0..12]: [191, 191, 191, 184, 194, 186, 191, 191, 184, 186, 186, 194]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265634392Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"[0]R generation[0..12]: [104, 104, 104, 102, 105, 103, 104, 104, 102, 103, 103, 105]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265645011Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265655939Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"[1]R flush_numbers[0..12]: [191, 191, 191, 184, 194, 186, 191, 191, 184, 186, 186, 194]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265666408Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"[1]R generation[0..12]: [104, 104, 104, 102, 105, 103, 104, 104, 102, 103, 103, 105]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265678606Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265689554Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"[2]R flush_numbers[0..12]: [191, 191, 191, 184, 194, 186, 191, 191, 184, 186, 186, 194]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265699993Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"[2]R generation[0..12]: [104, 104, 104, 102, 105, 103, 104, 104, 102, 103, 103, 105]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265710541Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.26572127Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Max found gen is 106","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265730938Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265740527Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Next flush: 195","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265749676Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265758894Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"":"downstairs","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265768703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265777752Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"client":"0","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.26578748Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"client":"1","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265797139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"client":"2","":"downstairs","session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265806847Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265815796Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"ce2921e2-8dfc-42ca-a54d-f67eab5206ec is now active with session: e7741e41-7e62-4cc3-a420-e8485ea83c0f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265825475Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} Wait for a query_work_queue command to finish before sending IO {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265839942Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.265847671Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561,"session_id":"e7741e41-7e62-4cc3-a420-e8485ea83c0f"} {"msg":"Activated sub_volume ce2921e2-8dfc-42ca-a54d-f67eab5206ec","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:10.26585652Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 435 len 16384 data: 13 10 10 13 02/30 Read block 495 len 8192 03/30 Read block 421 len 20480 04/30 Read block 540 len 8192 05/30 Write block 230 len 12288 data: 14 10 7 06/30 Read block 185 len 32768 07/30 Flush 08/30 Write block 464 len 20480 data: 5 4 8 10 13 09/30 Read block 40 len 40960 10/30 Write block 211 len 12288 data: 3 4 5 11/30 Read block 368 len 16384 12/30 Write block 244 len 4096 data: 10 13/30 Write block 401 len 8192 data: 5 6 14/30 Read block 198 len 36864 15/30 Write block 108 len 20480 data: 10 11 11 11 8 16/30 Flush 17/30 Flush 18/30 Flush 19/30 Write block 461 len 12288 data: 9 7 8 20/30 Flush 21/30 Write block 590 len 36864 data: 10 10 11 10 9 7 7 5 4 22/30 Write block 557 len 28672 data: 13 12 10 11 11 9 9 23/30 Read block 75 len 28672 24/30 Write block 436 len 12288 data: 11 11 14 25/30 Write block 573 len 24576 data: 8 9 11 13 10 10 26/30 Read block 403 len 12288 27/30 Write block 365 len 8192 data: 6 6 28/30 Read block 356 len 12288 29/30 Write block 485 len 16384 data: 6 5 5 4 30/30 Read block 455 len 4096 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Sent Done false 1001 Acked Read 2 Done Sent Done false 1002 Acked Read 5 Done Sent Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Write 2 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 9 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Read 7 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Write 4 Done Sent Sent false 1029 Acked Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 2 32 Done 30 0 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:10.274215138Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1561} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 191 191 191 F F F 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 102 102 102 184 184 184 F F F 4 080-099 105 105 105 194 194 194 F F F 5 100-119 106 103 106 196 186 196 F F F <--- 6 120-139 104 104 104 191 191 191 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 102 102 102 184 184 184 F F F 9 180-199 103 103 103 186 186 186 F F F 10 200-219 106 103 106 196 186 196 F F F <--- 11 220-239 106 105 106 195 194 195 F F F <--- 12 240-259 106 102 106 196 184 196 F F F <--- 13 260-279 102 102 102 184 184 184 F F F 14 280-299 103 103 103 186 186 186 F F F 15 300-319 103 103 103 186 186 186 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 104 104 104 191 191 191 F F F 18 360-379 98 98 98 179 179 179 T F T <--- 19 380-399 99 99 99 181 181 181 F F F 20 400-419 106 104 106 196 193 196 F F F <--- 21 420-439 106 102 106 195 184 195 T F T <--- 22 440-459 102 102 102 184 184 184 F F F 23 460-479 106 100 106 199 182 199 F F F <--- 24 480-499 101 101 101 183 183 183 T F T <--- 25 500-519 100 100 100 182 182 182 F F F 26 520-539 104 104 104 188 188 188 F F F 27 540-559 102 102 102 184 184 184 T F T <--- 28 560-579 105 105 105 194 194 194 T F T <--- 29 580-599 105 105 105 194 194 194 T F T <--- Max gen: 106, Max flush: 199 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 191 191 191 F F F 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 102 102 102 184 184 184 F F F 4 080-099 105 105 105 194 194 194 F F F 5 100-119 106 106 106 196 196 196 F F F 6 120-139 104 104 104 191 191 191 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 102 102 102 184 184 184 F F F 9 180-199 103 103 103 186 186 186 F F F 10 200-219 106 106 106 196 196 196 F F F 11 220-239 106 106 106 195 195 195 F F F 12 240-259 106 106 106 196 196 196 F F F 13 260-279 102 102 102 184 184 184 F F F 14 280-299 103 103 103 186 186 186 F F F 15 300-319 103 103 103 186 186 186 F F F 16 320-339 94 94 94 170 170 170 F F F 17 340-359 104 104 104 191 191 191 F F F 18 360-379 107 107 107 200 200 200 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 106 106 106 196 196 196 F F F 21 420-439 107 107 107 200 200 200 F F F 22 440-459 102 102 102 184 184 184 F F F 23 460-479 106 106 106 199 199 199 F F F 24 480-499 107 107 107 200 200 200 F F F 25 500-519 100 100 100 182 182 182 F F F 26 520-539 104 104 104 188 188 188 F F F 27 540-559 107 107 107 200 200 200 F F F 28 560-579 107 107 107 200 200 200 F F F 29 580-599 107 107 107 200 200 200 F F F Max gen: 107, Max flush: 200 Begin loop 54 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.52524369Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:14.532434659Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.532497699Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.539432596Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.539707576Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.539735621Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.53974407Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.539755709Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569} {"msg":"Crucible d521eeac-04b7-4b2c-a57b-573010c19039 has session id: de675288-93bd-44ba-81b0-d951142c58f5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540106187Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Upstairs opts: Upstairs UUID: d521eeac-04b7-4b2c-a57b-573010c19039, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540125524Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Crucible stats registered with UUID: d521eeac-04b7-4b2c-a57b-573010c19039","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540135453Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:14.540146301Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540304408Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"io task","client":"2","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540340713Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"io task","client":"0","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540351731Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"io task","client":"1","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540438999Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"task":"crutest"} {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540462745Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569} {"msg":"d521eeac-04b7-4b2c-a57b-573010c19039 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540489921Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.54049933Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"client":"0","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540509128Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"client":"1","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540518777Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"client":"2","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540528306Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"io task","client":"2","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540538134Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"io task","client":"0","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540548023Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"io task","client":"1","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:39708)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540602945Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"io task","client":"2","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:56159)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540718738Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"io task","client":"1","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:49756)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.540739725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"io task","client":"0","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541352155Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"client":"2","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541378601Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"client":"0","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541396319Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"client":"1","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541676548Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541686456Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"[0]R flush_numbers[0..12]: [191, 191, 191, 184, 194, 196, 191, 191, 184, 186, 196, 195]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541711023Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"[0]R generation[0..12]: [104, 104, 104, 102, 105, 106, 104, 104, 102, 103, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541722271Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.54173298Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"[1]R flush_numbers[0..12]: [191, 191, 191, 184, 194, 196, 191, 191, 184, 186, 196, 195]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541743548Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"[1]R generation[0..12]: [104, 104, 104, 102, 105, 106, 104, 104, 102, 103, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541753996Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541764765Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"[2]R flush_numbers[0..12]: [191, 191, 191, 184, 194, 196, 191, 191, 184, 186, 196, 195]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541775363Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"[2]R generation[0..12]: [104, 104, 104, 102, 105, 106, 104, 104, 102, 103, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541785812Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.54179664Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541807369Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541816977Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Next flush: 201","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541826226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541835465Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"":"downstairs","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541845433Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541854202Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"client":"0","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:14:14.54186402Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"client":"1","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541876759Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"client":"2","":"downstairs","session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541886567Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541895626Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"d521eeac-04b7-4b2c-a57b-573010c19039 is now active with session: de675288-93bd-44ba-81b0-d951142c58f5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541905344Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541914383Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541921982Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569,"session_id":"de675288-93bd-44ba-81b0-d951142c58f5"} {"msg":"Activated sub_volume d521eeac-04b7-4b2c-a57b-573010c19039","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:14.541931011Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 231 len 12288 data: 11 8 6 03/30 Read block 486 len 20480 04/30 Write block 330 len 8192 data: 9 11 05/30 Read block 581 len 32768 06/30 Write block 126 len 20480 data: 8 6 5 6 7 07/30 Flush 08/30 Write block 164 len 8192 data: 16 15 09/30 Write block 65 len 40960 data: 10 7 6 6 6 5 6 6 5 5 10/30 Read block 246 len 40960 11/30 Flush 12/30 Write block 426 len 8192 data: 8 8 13/30 Write block 336 len 36864 data: 9 7 8 8 9 12 13 11 10 14/30 Read block 478 len 28672 15/30 Write block 467 len 28672 data: 11 14 12 12 12 11 12 16/30 Write block 510 len 24576 data: 9 8 8 7 10 8 17/30 Write block 521 len 32768 data: 9 9 9 9 9 11 13 12 18/30 Read block 583 len 24576 19/30 Write block 99 len 32768 data: 14 14 13 10 11 8 9 9 20/30 Read block 410 len 32768 21/30 Write block 302 len 4096 data: 5 22/30 Write block 343 len 36864 data: 12 11 14 13 11 7 8 9 9 23/30 Flush 24/30 Write block 241 len 36864 data: 11 11 11 11 11 11 9 8 7 25/30 Read block 71 len 40960 26/30 Read block 346 len 28672 27/30 Write block 450 len 12288 data: 8 6 6 28/30 Write block 242 len 40960 data: 12 12 12 12 12 10 9 8 9 8 29/30 Write block 585 len 12288 data: 11 8 10 30/30 Read block 280 len 24576 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Sent Done Done false 1002 Acked Read 5 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Read 8 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Write 10 Sent Done Done false 1028 Acked Write 3 Sent Done Sent false 1029 Acked Read 6 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 2 31 Done 0 29 27 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:14.55123685Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1569} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 191 191 191 F F F 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 102 108 108 184 203 203 F F F <--- 4 080-099 105 108 108 194 204 204 F F F <--- 5 100-119 106 108 108 196 204 204 F F F <--- 6 120-139 104 108 108 191 202 202 F F F <--- 7 140-159 104 104 104 191 191 191 F F F 8 160-179 102 108 108 184 203 203 F F F <--- 9 180-199 103 103 103 186 186 186 F F F 10 200-219 106 106 106 196 196 196 F F F 11 220-239 106 108 108 195 202 202 F F F <--- 12 240-259 106 106 106 196 196 196 F T T <--- 13 260-279 102 102 102 184 184 184 F F F 14 280-299 103 103 103 186 186 186 F F F 15 300-319 103 108 108 186 204 204 F F F <--- 16 320-339 94 108 108 170 204 204 F F F <--- 17 340-359 104 108 108 191 204 204 F F F <--- 18 360-379 107 107 107 200 200 200 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 106 106 106 196 196 196 F F F 21 420-439 107 108 108 200 204 204 F F F <--- 22 440-459 102 102 102 184 184 184 F T T <--- 23 460-479 106 108 108 199 204 204 F F F <--- 24 480-499 107 107 107 200 200 200 F F F 25 500-519 100 108 108 182 204 204 F F F <--- 26 520-539 104 108 108 188 204 204 F F F <--- 27 540-559 107 107 107 200 200 200 F F F 28 560-579 107 107 107 200 200 200 F F F 29 580-599 107 107 107 200 200 200 F T T <--- Max gen: 108, Max flush: 204 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 191 191 191 F F F 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 108 108 108 203 203 203 F F F 4 080-099 108 108 108 204 204 204 F F F 5 100-119 108 108 108 204 204 204 F F F 6 120-139 108 108 108 202 202 202 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 108 108 108 203 203 203 F F F 9 180-199 103 103 103 186 186 186 F F F 10 200-219 106 106 106 196 196 196 F F F 11 220-239 108 108 108 202 202 202 F F F 12 240-259 109 109 109 205 205 205 F F F 13 260-279 102 102 102 184 184 184 F F F 14 280-299 103 103 103 186 186 186 F F F 15 300-319 108 108 108 204 204 204 F F F 16 320-339 108 108 108 204 204 204 F F F 17 340-359 108 108 108 204 204 204 F F F 18 360-379 107 107 107 200 200 200 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 106 106 106 196 196 196 F F F 21 420-439 108 108 108 204 204 204 F F F 22 440-459 109 109 109 205 205 205 F F F 23 460-479 108 108 108 204 204 204 F F F 24 480-499 107 107 107 200 200 200 F F F 25 500-519 108 108 108 204 204 204 F F F 26 520-539 108 108 108 204 204 204 F F F 27 540-559 107 107 107 200 200 200 F F F 28 560-579 107 107 107 200 200 200 F F F 29 580-599 109 109 109 205 205 205 F F F Max gen: 109, Max flush: 205 Begin loop 55 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.813339133Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.821744335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822060168Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822087904Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822096473Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822107801Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577} {"msg":"Crucible 71f64e3b-cf82-4d6b-ba62-98f0f576fc84 has session id: 662121af-a277-44d2-a1c4-5c4c7dff787e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822415986Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Upstairs opts: Upstairs UUID: 71f64e3b-cf82-4d6b-ba62-98f0f576fc84, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822452351Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Crucible stats registered with UUID: 71f64e3b-cf82-4d6b-ba62-98f0f576fc84","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822462969Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:18.822472118Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822633904Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"io task","client":"2","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822670329Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"io task","client":"0","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822681127Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"io task","client":"1","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822796541Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"task":"crutest"} {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822826896Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577} {"msg":"71f64e3b-cf82-4d6b-ba62-98f0f576fc84 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822840374Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822849333Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"client":"0","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822861381Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"client":"1","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.8228711Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"client":"2","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822882478Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"io task","client":"2","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822892447Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"io task","client":"0","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822902235Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"io task","client":"1","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"ds_connection connected from Ok(127.0.0.1:45923)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.822932761Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"io task","client":"2","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"ds_connection connected from Ok(127.0.0.1:54621)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.823030586Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"io task","client":"0","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"ds_connection connected from Ok(127.0.0.1:36414)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.823107225Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"io task","client":"1","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.823617691Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"client":"2","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.823635908Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"client":"0","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.823646986Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"client":"1","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.823953791Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.823978188Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"[0]R flush_numbers[0..12]: [191, 191, 191, 203, 204, 204, 202, 191, 203, 186, 196, 202]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824036129Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"[0]R generation[0..12]: [104, 104, 104, 108, 108, 108, 108, 104, 108, 103, 106, 108]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824048338Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824061416Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"[1]R flush_numbers[0..12]: [191, 191, 191, 203, 204, 204, 202, 191, 203, 186, 196, 202]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824072004Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"[1]R generation[0..12]: [104, 104, 104, 108, 108, 108, 108, 104, 108, 103, 106, 108]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824082413Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824093381Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"[2]R flush_numbers[0..12]: [191, 191, 191, 203, 204, 204, 202, 191, 203, 186, 196, 202]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824103899Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"[2]R generation[0..12]: [104, 104, 104, 108, 108, 108, 108, 104, 108, 103, 106, 108]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824114348Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824125066Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824134415Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824144254Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Next flush: 206","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824153642Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824163201Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"":"downstairs","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824173139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824181898Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,Wait for a query_work_queue command to finish before sending IO "client":"0","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824196096Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"client":"1","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824207094Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"client":"2","":"downstairs","session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824216823Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824225732Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"71f64e3b-cf82-4d6b-ba62-98f0f576fc84 is now active with session: 662121af-a277-44d2-a1c4-5c4c7dff787e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.82423559Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824244869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824252448Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577,"session_id":"662121af-a277-44d2-a1c4-5c4c7dff787e"} {"msg":"Activated sub_volume 71f64e3b-cf82-4d6b-ba62-98f0f576fc84","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:18.824261696Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 274 len 4096 data: 9 02/30 Write block 327 len 28672 data: 10 9 7 10 12 11 9 03/30 Read block 32 len 12288 04/30 Read block 268 len 20480 05/30 Read block 591 len 12288 06/30 Write block 365 len 20480 data: 7 7 5 5 6 07/30 Flush 08/30 Write block 407 len 4096 data: 7 09/30 Write block 327 len 20480 data: 11 10 8 11 13 10/30 Flush 11/30 Write block 325 len 32768 data: 7 7 12 11 9 12 14 12 12/30 Flush 13/30 Write block 499 len 12288 data: 9 9 7 14/30 Read block 168 len 40960 15/30 Write block 434 len 20480 data: 13 14 12 12 15 16/30 Write block 430 len 36864 data: 8 10 9 12 14 15 13 13 16 17/30 Write block 522 len 16384 data: 10 10 10 10 18/30 Read block 189 len 32768 19/30 Flush 20/30 Write block 468 len 32768 data: 15 13 13 13 12 13 12 9 21/30 Read block 131 len 40960 22/30 Write block 218 len 24576 data: 7 7 5 5 8 11 23/30 Read block 480 len 32768 24/30 Write block 111 len 16384 data: 12 9 7 9 25/30 Read block 302 len 36864 26/30 Read block 535 len 16384 27/30 Read block 82 len 8192 28/30 Read block 279 len 32768 29/30 Read block 40 len 40960 30/30 Write block 416 len 12288 data: 5 6 7 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:23 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1007 Acked Write 1 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Read 8 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Read 10 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 8 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 23 1 25 Done 22 0 22 44 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1006)) Some(JobId(1018)) Downstairs last five completed: 1006 1005 1004 1003 1002 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:18.833782865Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1577} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 191 191 191 F F F 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 108 108 108 203 203 203 F F F 4 080-099 108 108 108 204 204 204 F F F 5 100-119 108 108 108 204 204 204 T F T <--- 6 120-139 108 108 108 202 202 202 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 108 108 108 203 203 203 F F F 9 180-199 103 103 103 186 186 186 F F F 10 200-219 106 106 106 196 196 196 T F T <--- 11 220-239 108 108 108 202 202 202 T F T <--- 12 240-259 109 109 109 205 205 205 F F F 13 260-279 110 110 110 206 206 206 F F F 14 280-299 103 103 103 186 186 186 F F F 15 300-319 108 108 108 204 204 204 F F F 16 320-339 110 110 110 208 206 208 F F F <--- 17 340-359 108 108 108 204 204 204 F F F 18 360-379 110 110 110 206 206 206 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 110 106 110 207 196 207 T F T <--- 21 420-439 110 108 110 209 204 209 F F F <--- 22 440-459 109 109 109 205 205 205 F F F 23 460-479 108 108 108 204 204 204 T F T <--- 24 480-499 110 107 110 209 200 209 F F F <--- 25 500-519 110 108 110 209 204 209 F F F <--- 26 520-539 110 108 110 209 204 209 F F F <--- 27 540-559 107 107 107 200 200 200 F F F 28 560-579 107 107 107 200 200 200 F F F 29 580-599 109 109 109 205 205 205 F F F Max gen: 110, Max flush: 209 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 104 104 104 191 191 191 F F F 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 108 108 108 203 203 203 F F F 4 080-099 108 108 108 204 204 204 F F F 5 100-119 111 111 111 210 210 210 F F F 6 120-139 108 108 108 202 202 202 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 108 108 108 203 203 203 F F F 9 180-199 103 103 103 186 186 186 F F F 10 200-219 111 111 111 210 210 210 F F F 11 220-239 111 111 111 210 210 210 F F F 12 240-259 109 109 109 205 205 205 F F F 13 260-279 110 110 110 206 206 206 F F F 14 280-299 103 103 103 186 186 186 F F F 15 300-319 108 108 108 204 204 204 F F F 16 320-339 110 110 110 208 208 208 F F F 17 340-359 108 108 108 204 204 204 F F F 18 360-379 110 110 110 206 206 206 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 111 111 111 210 210 210 F F F 21 420-439 110 110 110 209 209 209 F F F 22 440-459 109 109 109 205 205 205 F F F 23 460-479 111 111 111 210 210 210 F F F 24 480-499 110 110 110 209 209 209 F F F 25 500-519 110 110 110 209 209 209 F F F 26 520-539 110 110 110 209 209 209 F F F 27 540-559 107 107 107 200 200 200 F F F 28 560-579 107 107 107 200 200 200 F F F 29 580-599 109 109 109 205 205 205 F F F Max gen: 111, Max flush: 210 Begin loop 56 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.083719335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.091819921Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092121647Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092152193Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092160612Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.09217214Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585} {"msg":"Crucible 49c673e1-0388-4193-aaff-83dda639ee6b has session id: 1ecc5318-6de6-432e-a31a-6df6874bd298","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092493183Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Upstairs opts: Upstairs UUID: 49c673e1-0388-4193-aaff-83dda639ee6b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092523449Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Crucible stats registered with UUID: 49c673e1-0388-4193-aaff-83dda639ee6b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092533737Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:23.092544976Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092683915Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"io task","client":"0","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092706112Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"io task","client":"2","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092716001Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"io task","client":"1","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092883466Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"task":"crutest"} {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092931369Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585} {"msg":"49c673e1-0388-4193-aaff-83dda639ee6b active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092940218Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092948926Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"client":"0","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092958725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"client":"1","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092968304Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"client":"2","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092977792Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"io task","client":"2","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.092987751Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"io task","client":"0","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.093013447Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"io task","client":"1","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"ds_connection connected from Ok(127.0.0.1:46224)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.093043793Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"io task","client":"2","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"ds_connection connected from Ok(127.0.0.1:42164)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.093081037Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"io task","client":"1","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"ds_connection connected from Ok(127.0.0.1:45102)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.093095015Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"io task","client":"0","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.093750349Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"client":"2","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.093772036Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"client":"0","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.093783055Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"client":"1","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094074082Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094084221Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"[0]R flush_numbers[0..12]: [191, 191, 191, 203, 204, 210, 202, 191, 203, 186, 210, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.0941564Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"[0]R generation[0..12]: [104, 104, 104, 108, 108, 111, 108, 104, 108, 103, 111, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094169588Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094180626Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"[1]R flush_numbers[0..12]: [191, 191, 191, 203, 204, 210, 202, 191, 203, 186, 210, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094191285Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"[1]R generation[0..12]: [104, 104, 104, 108, 108, 111, 108, 104, 108, 103, 111, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094201653Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094212402Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"[2]R flush_numbers[0..12]: [191, 191, 191, 203, 204, 210, 202, 191, 203, 186, 210, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.09422287Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"[2]R generation[0..12]: [104, 104, 104, 108, 108, 111, 108, 104, 108, 103, 111, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094233439Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094244267Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Max found gen is 112","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094255595Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Generation requested: 112 >= found:112","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094265314Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Next flush: 211","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094274533Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094283691Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"":"downstairs","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.0942936Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094302309Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"client":"0","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094311907Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"client":"1","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094321506Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"client":"2","":"downstairs","session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094331374Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094340423Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"49c673e1-0388-4193-aaff-83dda639ee6b is now active with session: 1ecc5318-6de6-432e-a31a-6df6874bd298","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-04-15T18:14:23.094350322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.09436358Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094371349Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585,"session_id":"1ecc5318-6de6-432e-a31a-6df6874bd298"} {"msg":"Activated sub_volume 49c673e1-0388-4193-aaff-83dda639ee6b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:23.094380497Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 365 len 8192 data: 8 8 03/30 Write block 9 len 32768 data: 9 9 10 10 11 11 10 8 04/30 Read block 215 len 28672 05/30 Read block 385 len 8192 06/30 Write block 570 len 24576 data: 10 11 11 9 10 12 07/30 Read block 546 len 32768 08/30 Flush 09/30 Write block 536 len 12288 data: 13 14 12 10/30 Write block 449 len 32768 data: 11 9 7 7 7 10 11 13 11/30 Write block 353 len 12288 data: 11 11 11 12/30 Write block 332 len 20480 data: 13 10 10 9 10 13/30 Write block 448 len 32768 data: 10 12 10 8 8 8 11 12 14/30 Write block 458 len 32768 data: 15 13 11 10 8 9 6 5 15/30 Read block 137 len 16384 16/30 Write block 471 len 32768 data: 14 13 14 13 10 7 5 4 17/30 Write block 281 len 16384 data: 9 6 8 9 18/30 Write block 203 len 32768 data: 6 10 8 7 6 4 4 3 19/30 Read block 546 len 16384 20/30 Write block 110 len 20480 data: 12 13 10 8 10 21/30 Write block 569 len 16384 data: 9 11 12 12 22/30 Read block 12 len 8192 23/30 Read block 45 len 32768 24/30 Write block 304 len 36864 data: 8 8 11 7 8 11 10 13 11 25/30 Read block 175 len 32768 26/30 Read block 216 len 24576 27/30 Read block 459 len 12288 28/30 Write block 550 len 8192 data: 8 10 29/30 Read block 432 len 28672 30/30 Read block 221 len 20480 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 2 Done Done Done false 1002 Acked Write 8 Done Sent Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Read 2 Done Sent Done false 1005 Acked Write 6 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Read 4 Done Sent Done false 1015 Acked Write 8 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Write 4 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Read 3 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 1 29 Done 29 1 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) Some(JobId(1000)) Some(JobId(1007)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:23.102703361Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1585} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 104 112 212 191 212 F F F <--- 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 108 108 108 203 203 203 F F F 4 080-099 108 108 108 204 204 204 F F F 5 100-119 111 111 111 210 210 210 T F T <--- 6 120-139 108 108 108 202 202 202 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 108 108 108 203 203 203 F F F 9 180-199 103 103 103 186 186 186 F F F 10 200-219 111 111 111 210 210 210 T F T <--- 11 220-239 111 111 111 210 210 210 F F F 12 240-259 109 109 109 205 205 205 F F F 13 260-279 110 110 110 206 206 206 F F F 14 280-299 103 103 103 186 186 186 T F T <--- 15 300-319 108 108 108 204 204 204 T F T <--- 16 320-339 110 110 110 208 208 208 T F T <--- 17 340-359 108 108 108 204 204 204 T F T <--- 18 360-379 112 110 112 212 206 212 F T F <--- 19 380-399 99 99 99 181 181 181 F F F 20 400-419 111 111 111 210 210 210 F F F 21 420-439 110 110 110 209 209 209 F F F 22 440-459 109 109 109 205 205 205 T F T <--- 23 460-479 111 111 111 210 210 210 T F T <--- 24 480-499 110 110 110 209 209 209 F F F 25 500-519 110 110 110 209 209 209 F F F 26 520-539 110 110 110 209 209 209 T F T <--- 27 540-559 107 107 107 200 200 200 T F T <--- 28 560-579 112 107 112 212 200 212 T F T <--- 29 580-599 109 109 109 205 205 205 F F F Max gen: 112, Max flush: 212 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 112 112 212 212 212 F F F 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 108 108 108 203 203 203 F F F 4 080-099 108 108 108 204 204 204 F F F 5 100-119 113 113 113 213 213 213 F F F 6 120-139 108 108 108 202 202 202 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 108 108 108 203 203 203 F F F 9 180-199 103 103 103 186 186 186 F F F 10 200-219 113 113 113 213 213 213 F F F 11 220-239 111 111 111 210 210 210 F F F 12 240-259 109 109 109 205 205 205 F F F 13 260-279 110 110 110 206 206 206 F F F 14 280-299 113 113 113 213 213 213 F F F 15 300-319 113 113 113 213 213 213 F F F 16 320-339 113 113 113 213 213 213 F F F 17 340-359 113 113 113 213 213 213 F F F 18 360-379 112 112 112 212 212 212 F F F 19 380-399 99 99 99 181 181 181 F F F 20 400-419 111 111 111 210 210 210 F F F 21 420-439 110 110 110 209 209 209 F F F 22 440-459 113 113 113 213 213 213 F F F 23 460-479 113 113 113 213 213 213 F F F 24 480-499 110 110 110 209 209 209 F F F 25 500-519 110 110 110 209 209 209 F F F 26 520-539 113 113 113 213 213 213 F F F 27 540-559 113 113 113 213 213 213 F F F 28 560-579 113 113 113 213 213 213 F F F 29 580-599 109 109 109 205 205 205 F F F Max gen: 113, Max flush: 213 Begin loop 57 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.356262272Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.364343641Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.36462379Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.364653706Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.364662734Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.364674243Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593} {"msg":"Crucible 342d7ccd-f865-4288-8ef4-cf1628f30fb3 has session id: 4800995a-854c-493a-b2f4-d6ec033e7725","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.36503202Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Upstairs opts: Upstairs UUID: 342d7ccd-f865-4288-8ef4-cf1628f30fb3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365060386Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Crucible stats registered with UUID: 342d7ccd-f865-4288-8ef4-cf1628f30fb3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365071205Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:27.365080373Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365230211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"io task","client":"1","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365252688Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"io task","client":"0","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365262557Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"io task","client":"2","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365399507Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"task":"crutest"} {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365422703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593} {"msg":"342d7ccd-f865-4288-8ef4-cf1628f30fb3 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365441511Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365450689Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"client":"0","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365462637Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"client":"1","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365472156Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"client":"2","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365481585Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"io task","client":"2","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365491403Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"io task","client":"0","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365509621Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"io task","client":"1","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"ds_connection connected from Ok(127.0.0.1:55438)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365603727Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"io task","client":"0","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"ds_connection connected from Ok(127.0.0.1:35788)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365679696Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"io task","client":"1","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"ds_connection connected from Ok(127.0.0.1:50576)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.365691714Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"io task","client":"2","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366325321Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"client":"1","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366350528Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"client":"0","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366362226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"client":"2","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366637026Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366646434Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"[0]R flush_numbers[0..12]: [212, 191, 191, 203, 204, 213, 202, 191, 203, 186, 213, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.36667881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"[0]R generation[0..12]: [112, 104, 104, 108, 108, 113, 108, 104, 108, 103, 113, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366690178Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366703066Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"[1]R flush_numbers[0..12]: [212, 191, 191, 203, 204, 213, 202, 191, 203, 186, 213, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366713655Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"[1]R generation[0..12]: [112, 104, 104, 108, 108, 113, 108, 104, 108, 103, 113, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366724053Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366734852Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"[2]R flush_numbers[0..12]: [212, 191, 191, 203, 204, 213, 202, 191, 203, 186, 213, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.36674537Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"[2]R generation[0..12]: [112, 104, 104, 108, 108, 113, 108, 104, 108, 103, 113, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366755618Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366766347Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366775656Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366785304Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Next flush: 214","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366794513Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366803761Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"":"downstairs","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.36681349Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366822219Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"client":"0","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366831927Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"client":"1","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366842446Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"client":"2","":"downstairs","session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} Wait for a query_work_queue command to finish before sending IO {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366856594Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366865752Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"342d7ccd-f865-4288-8ef4-cf1628f30fb3 is now active with session: 4800995a-854c-493a-b2f4-d6ec033e7725","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366875521Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.36688443Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366892209Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593,"session_id":"4800995a-854c-493a-b2f4-d6ec033e7725"} {"msg":"Activated sub_volume 342d7ccd-f865-4288-8ef4-cf1628f30fb3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:27.366901317Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 89 len 8192 data: 12 10 02/30 Read block 491 len 24576 03/30 Write block 339 len 16384 data: 9 10 13 14 04/30 Write block 374 len 16384 data: 10 8 11 11 05/30 Read block 110 len 28672 06/30 Read block 542 len 40960 07/30 Write block 383 len 8192 data: 8 6 08/30 Write block 168 len 24576 data: 9 10 8 9 9 7 09/30 Read block 436 len 20480 10/30 Read block 496 len 8192 11/30 Read block 108 len 40960 12/30 Read block 572 len 28672 13/30 Write block 212 len 12288 data: 5 6 4 14/30 Read block 189 len 32768 15/30 Read block 26 len 32768 16/30 Read block 311 len 32768 17/30 Flush 18/30 Write block 198 len 8192 data: 10 10 19/30 Read block 167 len 8192 20/30 Read block 403 len 4096 21/30 Read block 67 len 20480 22/30 Read block 91 len 28672 23/30 Read block 63 len 20480 24/30 Write block 194 len 20480 data: 11 11 10 11 11 25/30 Read block 215 len 40960 26/30 Write block 277 len 12288 data: 6 4 9 27/30 Write block 127 len 24576 data: 7 6 7 8 8 10 28/30 Read block 151 len 4096 29/30 Read block 131 len 32768 30/30 Write block 432 len 4096 data: 10 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 6 Done Sent Done false 1002 Acked Write 4 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 7 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Write 6 Done Sent Done false 1008 Acked Read 5 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Read 10 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Write 5 Done Sent Done false 1024 Acked Read 10 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) None Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:27.375361391Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1593} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 112 112 212 212 212 F F F 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 108 108 108 203 203 203 F F F 4 080-099 114 108 114 214 204 214 F T F <--- 5 100-119 113 113 113 213 213 213 F F F 6 120-139 108 108 108 202 202 202 T F T <--- 7 140-159 104 104 104 191 191 191 F F F 8 160-179 114 108 114 214 203 214 F F F <--- 9 180-199 103 103 103 186 186 186 T F T <--- 10 200-219 114 113 114 214 213 214 F F F <--- 11 220-239 111 111 111 210 210 210 F F F 12 240-259 109 109 109 205 205 205 F F F 13 260-279 110 110 110 206 206 206 T F T <--- 14 280-299 113 113 113 213 213 213 F F F 15 300-319 113 113 113 213 213 213 F F F 16 320-339 114 113 114 214 213 214 F F F <--- 17 340-359 114 113 114 214 213 214 F F F <--- 18 360-379 114 112 114 214 212 214 F F F <--- 19 380-399 114 99 114 214 181 214 F F F <--- 20 400-419 111 111 111 210 210 210 F F F 21 420-439 110 110 110 209 209 209 T F T <--- 22 440-459 113 113 113 213 213 213 F F F 23 460-479 113 113 113 213 213 213 F F F 24 480-499 110 110 110 209 209 209 F F F 25 500-519 110 110 110 209 209 209 F F F 26 520-539 113 113 113 213 213 213 F F F 27 540-559 113 113 113 213 213 213 F F F 28 560-579 113 113 113 213 213 213 F F F 29 580-599 109 109 109 205 205 205 F F F Max gen: 114, Max flush: 214 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 57 Verifying data now No change after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 112 112 112 212 212 212 F F F 1 020-039 104 104 104 191 191 191 F F F 2 040-059 104 104 104 191 191 191 F F F 3 060-079 108 108 108 203 203 203 F F F 4 080-099 114 114 114 214 214 214 F F F 5 100-119 113 113 113 213 213 213 F F F 6 120-139 115 115 115 215 215 215 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 114 114 114 214 214 214 F F F 9 180-199 115 115 115 215 215 215 F F F 10 200-219 114 114 114 214 214 214 F F F 11 220-239 111 111 111 210 210 210 F F F 12 240-259 109 109 109 205 205 205 F F F 13 260-279 115 115 115 215 215 215 F F F 14 280-299 113 113 113 213 213 213 F F F 15 300-319 113 113 113 213 213 213 F F F 16 320-339 114 114 114 214 214 214 F F F 17 340-359 114 114 114 214 214 214 F F F 18 360-379 114 114 114 214 214 214 F F F 19 380-399 114 114 114 214 214 214 F F F 20 400-419 111 111 111 210 210 210 F F F 21 420-439 115 115 115 215 215 215 F F F 22 440-459 113 113 113 213 213 213 F F F 23 460-479 113 113 113 213 213 213 F F F 24 480-499 110 110 110 209 209 209 F F F 25 500-519 110 110 110 209 209 209 F F F 26 520-539 113 113 113 213 213 213 F F F 27 540-559 113 113 113 213 213 213 F F F 28 560-579 113 113 113 213 213 213 F F F 29 580-599 109 109 109 205 205 205 F F F Max gen: 115, Max flush: 215 Begin loop 58 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.622206173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.630286612Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.630572681Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.630599427Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.630608125Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.630619754Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601} {"msg":"Crucible d9b86e15-ec3e-405a-8318-03bca1be17fc has session id: 8f46ddcf-701a-4daa-93bb-b87e3e169d4f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.630942756Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Upstairs opts: Upstairs UUID: d9b86e15-ec3e-405a-8318-03bca1be17fc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631024534Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Crucible stats registered with UUID: d9b86e15-ec3e-405a-8318-03bca1be17fc","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631038422Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:31.631047561Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631205548Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"io task","client":"0","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631228175Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"io task","client":"2","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631238173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"io task","client":"1","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631360585Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"task":"crutest"} {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631378673Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601} {"msg":"d9b86e15-ec3e-405a-8318-03bca1be17fc active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631413668Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631423116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"client":"0","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631432785Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"client":"1","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631442323Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"client":"2","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631451942Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"io task","client":"2","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631461981Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"io task","client":"0","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631471739Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"io task","client":"1","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"ds_connection connected from Ok(127.0.0.1:34450)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.63153661Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"io task","client":"2","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"ds_connection connected from Ok(127.0.0.1:43312)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631609139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"io task","client":"1","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"ds_connection connected from Ok(127.0.0.1:33252)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.631622187Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"io task","client":"0","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632306827Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"client":"1","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632330054Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"client":"2","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632341342Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"client":"0","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632624691Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632657666Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"[0]R flush_numbers[0..12]: [212, 191, 191, 203, 214, 213, 215, 191, 214, 215, 214, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632722056Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"[0]R generation[0..12]: [112, 104, 104, 108, 114, 113, 115, 104, 114, 115, 114, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632733355Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632744223Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"[1]R flush_numbers[0..12]: [212, 191, 191, 203, 214, 213, 215, 191, 214, 215, 214, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632754842Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"[1]R generation[0..12]: [112, 104, 104, 108, 114, 113, 115, 104, 114, 115, 114, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.63276524Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632776238Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"[2]R flush_numbers[0..12]: [212, 191, 191, 203, 214, 213, 215, 191, 214, 215, 214, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632786767Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"[2]R generation[0..12]: [112, 104, 104, 108, 114, 113, 115, 104, 114, 115, 114, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632797185Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632809774Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Max found gen is 116","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632819132Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Generation requested: 116 >= found:116","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632828751Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Next flush: 216","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632838019Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632852417Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"":"downstairs","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632862066Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632870785Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"client":"0","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632880333Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"client":"1","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-04-15T18:14:31.632889962Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"client":"2","":"downstairs","session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.63290395Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632913008Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"d9b86e15-ec3e-405a-8318-03bca1be17fc is now active with session: 8f46ddcf-701a-4daa-93bb-b87e3e169d4f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632922827Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632931906Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632939465Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601,"session_id":"8f46ddcf-701a-4daa-93bb-b87e3e169d4f"} {"msg":"Activated sub_volume d9b86e15-ec3e-405a-8318-03bca1be17fc","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:31.632948503Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 10 len 20480 data: 10 11 11 12 12 02/30 Read block 457 len 4096 03/30 Read block 415 len 16384 04/30 Flush 05/30 Write block 22 len 4096 data: 4 06/30 Read block 425 len 32768 07/30 Flush 08/30 Write block 313 len 20480 data: 13 11 11 14 14 09/30 Write block 433 len 16384 data: 13 15 16 14 10/30 Write block 60 len 12288 data: 6 8 8 11/30 Flush 12/30 Write block 123 len 36864 data: 9 8 7 9 8 7 8 9 9 13/30 Read block 150 len 4096 14/30 Write block 379 len 8192 data: 9 8 15/30 Read block 553 len 28672 16/30 Write block 187 len 20480 data: 7 6 6 9 10 17/30 Write block 582 len 8192 data: 7 8 18/30 Read block 503 len 36864 19/30 Read block 123 len 20480 20/30 Read block 481 len 28672 21/30 Write block 56 len 20480 data: 11 10 9 8 7 22/30 Read block 369 len 28672 23/30 Write block 519 len 12288 data: 8 8 10 24/30 Write block 279 len 32768 data: 10 9 10 7 9 10 9 8 25/30 Read block 286 len 16384 26/30 Write block 572 len 32768 data: 13 10 11 13 14 11 11 10 27/30 Read block 523 len 16384 28/30 Write block 432 len 36864 data: 11 14 16 17 15 14 17 12 11 29/30 Read block 49 len 36864 30/30 Write block 455 len 16384 data: 13 14 14 16 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Sent false 1001 Acked Read 1 Done Done Sent false 1002 Acked Read 4 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 1 Done Done Sent false 1005 Acked Read 8 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:31.642320154Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1601} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 112 216 216 212 F F F <--- 1 020-039 116 116 104 217 217 191 F F F <--- 2 040-059 104 104 104 191 191 191 T T F <--- 3 060-079 116 116 108 218 218 203 T T F <--- 4 080-099 114 114 114 214 214 214 F F F 5 100-119 113 113 113 213 213 213 F F F 6 120-139 115 115 115 215 215 215 T T F <--- 7 140-159 104 104 104 191 191 191 F F F 8 160-179 114 114 114 214 214 214 F F F 9 180-199 115 115 115 215 215 215 T T F <--- 10 200-219 114 114 114 214 214 214 F F F 11 220-239 111 111 111 210 210 210 F F F 12 240-259 109 109 109 205 205 205 F F F 13 260-279 115 115 115 215 215 215 T T F <--- 14 280-299 113 113 113 213 213 213 T T F <--- 15 300-319 116 116 113 218 218 213 F F F <--- 16 320-339 114 114 114 214 214 214 F F F 17 340-359 114 114 114 214 214 214 F F F 18 360-379 114 114 114 214 214 214 T T F <--- 19 380-399 114 114 114 214 214 214 T T F <--- 20 400-419 111 111 111 210 210 210 F F F 21 420-439 116 116 115 218 218 215 T T F <--- 22 440-459 113 113 113 213 213 213 T T F <--- 23 460-479 113 113 113 213 213 213 F F F 24 480-499 110 110 110 209 209 209 F F F 25 500-519 110 110 110 209 209 209 T T F <--- 26 520-539 113 113 113 213 213 213 T T F <--- 27 540-559 113 113 113 213 213 213 F F F 28 560-579 113 113 113 213 213 213 T T F <--- 29 580-599 109 109 109 205 205 205 T T F <--- Max gen: 116, Max flush: 218 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 58 Verifying data now No change after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 116 116 116 216 216 216 F F F 1 020-039 116 116 116 217 217 217 F F F 2 040-059 117 117 117 219 219 219 F F F 3 060-079 117 117 117 219 219 219 F F F 4 080-099 114 114 114 214 214 214 F F F 5 100-119 113 113 113 213 213 213 F F F 6 120-139 117 117 117 219 219 219 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 114 114 114 214 214 214 F F F 9 180-199 117 117 117 219 219 219 F F F 10 200-219 114 114 114 214 214 214 F F F 11 220-239 111 111 111 210 210 210 F F F 12 240-259 109 109 109 205 205 205 F F F 13 260-279 117 117 117 219 219 219 F F F 14 280-299 117 117 117 219 219 219 F F F 15 300-319 116 116 116 218 218 218 F F F 16 320-339 114 114 114 214 214 214 F F F 17 340-359 114 114 114 214 214 214 F F F 18 360-379 117 117 117 219 219 219 F F F 19 380-399 117 117 117 219 219 219 F F F 20 400-419 111 111 111 210 210 210 F F F 21 420-439 117 117 117 219 219 219 F F F 22 440-459 117 117 117 219 219 219 F F F 23 460-479 113 113 113 213 213 213 F F F 24 480-499 110 110 110 209 209 209 F F F 25 500-519 117 117 117 219 219 219 F F F 26 520-539 117 117 117 219 219 219 F F F 27 540-559 113 113 113 213 213 213 F F F 28 560-579 117 117 117 219 219 219 F F F 29 580-599 117 117 117 219 219 219 F F F Max gen: 117, Max flush: 219 Begin loop 59 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.906577051Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.914660439Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.914937819Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.914963675Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.914972184Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.914983592Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609} {"msg":"Crucible bc6ca533-5cc8-400e-9845-bf913ad2294f has session id: 27da0768-427f-4249-a959-9e0d750b7d8d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915288318Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Upstairs opts: Upstairs UUID: bc6ca533-5cc8-400e-9845-bf913ad2294f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915316343Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Crucible stats registered with UUID: bc6ca533-5cc8-400e-9845-bf913ad2294f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915327492Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:35.915336691Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915490068Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"io task","client":"0","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915513165Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"io task","client":"1","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915523263Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"io task","client":"2","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915645755Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"task":"crutest"} {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915668932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609} {"msg":"bc6ca533-5cc8-400e-9845-bf913ad2294f active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915708996Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915718745Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"client":"0","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915729223Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"client":"1","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915739162Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"client":"2","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.91574887Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"io task","client":"0","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915759189Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"io task","client":"2","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915813481Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"io task","client":"1","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:58625)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915824479Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"io task","client":"2","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:47828)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915834778Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"io task","client":"0","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:61038)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.915903428Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"io task","client":"1","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916463306Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"client":"0","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916486412Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"client":"1","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.91657444Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"client":"2","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916769381Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916788558Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"[0]R flush_numbers[0..12]: [216, 217, 219, 219, 214, 213, 219, 191, 214, 219, 214, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916836101Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"[0]R generation[0..12]: [116, 116, 117, 117, 114, 113, 117, 104, 114, 117, 114, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916849649Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916860648Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"[1]R flush_numbers[0..12]: [216, 217, 219, 219, 214, 213, 219, 191, 214, 219, 214, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916871236Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"[1]R generation[0..12]: [116, 116, 117, 117, 114, 113, 117, 104, 114, 117, 114, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916881605Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916892493Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"[2]R flush_numbers[0..12]: [216, 217, 219, 219, 214, 213, 219, 191, 214, 219, 214, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916902982Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"[2]R generation[0..12]: [116, 116, 117, 117, 114, 113, 117, 104, 114, 117, 114, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.91691319Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916924068Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Max found gen is 118","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916933397Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Generation requested: 118 >= found:118","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916942876Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Next flush: 220","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916952104Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916961303Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"":"downstairs","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916970882Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.91697949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"client":"0","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916990029Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"client":"1","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.916999617Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"client":"2","":"downstairs","session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.917009496Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.917018485Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"Wait for a query_work_queue command to finish before sending IO session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"bc6ca533-5cc8-400e-9845-bf913ad2294f is now active with session: 27da0768-427f-4249-a959-9e0d750b7d8d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.917031953Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.917040791Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.91704823Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609,"session_id":"27da0768-427f-4249-a959-9e0d750b7d8d"} {"msg":"Activated sub_volume bc6ca533-5cc8-400e-9845-bf913ad2294f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:35.917057219Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 12 len 28672 data: 12 13 13 11 9 9 9 02/30 Read block 439 len 12288 03/30 Read block 478 len 16384 04/30 Read block 226 len 40960 05/30 Write block 577 len 32768 data: 12 12 11 9 8 8 9 10 06/30 Read block 440 len 32768 07/30 Read block 315 len 16384 08/30 Read block 257 len 40960 09/30 Write block 384 len 24576 data: 7 5 5 7 9 9 10/30 Read block 128 len 16384 11/30 Read block 249 len 20480 12/30 Write block 356 len 8192 data: 12 12 13/30 Write block 377 len 12288 data: 12 8 10 14/30 Write block 163 len 24576 data: 16 17 16 13 13 10 15/30 Read block 544 len 16384 16/30 Flush 17/30 Write block 384 len 12288 data: 8 6 6 18/30 Write block 121 len 4096 data: 10 19/30 Read block 534 len 36864 20/30 Write block 404 len 16384 data: 6 7 8 8 21/30 Read block 330 len 40960 22/30 Read block 337 len 36864 23/30 Write block 64 len 16384 data: 10 11 8 7 24/30 Read block 307 len 36864 25/30 Read block 0 len 16384 26/30 Read block 104 len 24576 27/30 Read block 241 len 36864 28/30 Write block 130 len 20480 data: 10 10 11 10 8 29/30 Read block 441 len 8192 30/30 Read block 165 len 4096 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Read 10 Done Done Done false 1004 Acked Write 8 Done Done Sent false 1005 Acked Read 8 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Read 10 Done Done Sent false 1008 Acked Write 6 Done Done Sent false 1009 Acked Read 4 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Write 3 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Write 1 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Read 10 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Write 4 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Read 6 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Read 2 Done Done Sent false 1029 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 26 26 Done 30 30 4 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1015)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:35.926550012Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1609} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 116 220 220 216 F F T <--- 1 020-039 116 116 116 217 217 217 F F F 2 040-059 117 117 117 219 219 219 F F F 3 060-079 117 117 117 219 219 219 T T F <--- 4 080-099 114 114 114 214 214 214 F F F 5 100-119 113 113 113 213 213 213 F F F 6 120-139 117 117 117 219 219 219 T T F <--- 7 140-159 104 104 104 191 191 191 F F F 8 160-179 118 118 114 220 220 214 F F F <--- 9 180-199 117 117 117 219 219 219 F F F 10 200-219 114 114 114 214 214 214 F F F 11 220-239 111 111 111 210 210 210 F F F 12 240-259 109 109 109 205 205 205 F F F 13 260-279 117 117 117 219 219 219 F F F 14 280-299 117 117 117 219 219 219 F F F 15 300-319 116 116 116 218 218 218 F F F 16 320-339 114 114 114 214 214 214 F F F 17 340-359 118 118 114 220 220 214 F F F <--- 18 360-379 118 118 117 220 220 219 F F F <--- 19 380-399 118 118 117 220 220 219 T T F <--- 20 400-419 111 111 111 210 210 210 T T F <--- 21 420-439 117 117 117 219 219 219 F F F 22 440-459 117 117 117 219 219 219 F F F 23 460-479 113 113 113 213 213 213 F F F 24 480-499 110 110 110 209 209 209 F F F 25 500-519 117 117 117 219 219 219 F F F 26 520-539 117 117 117 219 219 219 F F F 27 540-559 113 113 113 213 213 213 F F F 28 560-579 118 118 117 220 220 219 F F F <--- 29 580-599 118 118 117 220 220 219 F F F <--- Max gen: 118, Max flush: 220 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 59 Verifying data now No change after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 220 220 220 F F F 1 020-039 116 116 116 217 217 217 F F F 2 040-059 117 117 117 219 219 219 F F F 3 060-079 119 119 119 221 221 221 F F F 4 080-099 114 114 114 214 214 214 F F F 5 100-119 113 113 113 213 213 213 F F F 6 120-139 119 119 119 221 221 221 F F F 7 140-159 104 104 104 191 191 191 F F F 8 160-179 118 118 118 220 220 220 F F F 9 180-199 117 117 117 219 219 219 F F F 10 200-219 114 114 114 214 214 214 F F F 11 220-239 111 111 111 210 210 210 F F F 12 240-259 109 109 109 205 205 205 F F F 13 260-279 117 117 117 219 219 219 F F F 14 280-299 117 117 117 219 219 219 F F F 15 300-319 116 116 116 218 218 218 F F F 16 320-339 114 114 114 214 214 214 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 118 118 118 220 220 220 F F F 19 380-399 119 119 119 221 221 221 F F F 20 400-419 119 119 119 221 221 221 F F F 21 420-439 117 117 117 219 219 219 F F F 22 440-459 117 117 117 219 219 219 F F F 23 460-479 113 113 113 213 213 213 F F F 24 480-499 110 110 110 209 209 209 F F F 25 500-519 117 117 117 219 219 219 F F F 26 520-539 117 117 117 219 219 219 F F F 27 540-559 113 113 113 213 213 213 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 118 118 118 220 220 220 F F F Max gen: 119, Max flush: 221 Begin loop 60 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.175055001Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.183176084Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.183463013Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.183488989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.183497507Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.183509056Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617} {"msg":"Crucible a294c5a0-034d-4bc9-9c0f-6ef78d501c69 has session id: d7fd2492-04ef-4e18-8934-9ccada97c5bc","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.183860324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Upstairs opts: Upstairs UUID: a294c5a0-034d-4bc9-9c0f-6ef78d501c69, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.183912937Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Crucible stats registered with UUID: a294c5a0-034d-4bc9-9c0f-6ef78d501c69","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.183923755Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:40.183932674Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184087881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"io task","client":"2","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184124466Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"io task","client":"0","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184135324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"io task","client":"1","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184243578Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"task":"crutest"} {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184260596Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617} {"msg":"a294c5a0-034d-4bc9-9c0f-6ef78d501c69 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.18429814Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184307789Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"client":"0","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184317628Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"client":"1","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184327406Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"client":"2","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184336945Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"io task","client":"2","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.18436892Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"io task","client":"1","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184389567Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"io task","client":"0","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:64799)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184419203Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"io task","client":"1","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:61098)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184432131Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"io task","client":"2","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"ds_connection connected from Ok(127.0.0.1:62510)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.184462166Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"io task","client":"0","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185076077Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"client":"0","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185105362Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"client":"1","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.18526158Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"client":"2","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185357866Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185382372Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"[0]R flush_numbers[0..12]: [220, 217, 219, 221, 214, 213, 221, 191, 220, 219, 214, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185584852Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"[0]R generation[0..12]: [118, 116, 117, 119, 114, 113, 119, 104, 118, 117, 114, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185595961Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185606989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"[1]R flush_numbers[0..12]: [220, 217, 219, 221, 214, 213, 221, 191, 220, 219, 214, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185617448Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"[1]R generation[0..12]: [118, 116, 117, 119, 114, 113, 119, 104, 118, 117, 114, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185627846Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185638764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"[2]R flush_numbers[0..12]: [220, 217, 219, 221, 214, 213, 221, 191, 220, 219, 214, 210]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185649273Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"[2]R generation[0..12]: [118, 116, 117, 119, 114, 113, 119, 104, 118, 117, 114, 111]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185661881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.1856727Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Max found gen is 120","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185684958Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185694566Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Next flush: 222","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185703685Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185712954Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"":"downstairs","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185722542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185731211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"client":"0","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.18574094Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"client":"1","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185750408Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"client":"2","":"downstairs","session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185760027Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185768935Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"a294c5a0-034d-4bc9-9c0f-6ef78d501c69 is now active with session: d7fd2492-04ef-4e18-8934-9ccada97c5bc","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185778674Z"Wait for a query_work_queue command to finish before sending IO ,"hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185792392Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.185799961Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617,"session_id":"d7fd2492-04ef-4e18-8934-9ccada97c5bc"} {"msg":"Activated sub_volume a294c5a0-034d-4bc9-9c0f-6ef78d501c69","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:40.18580897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 494 len 8192 data: 7 8 02/30 Write block 269 len 20480 data: 13 10 11 11 11 03/30 Flush 04/30 Write block 87 len 24576 data: 13 12 13 11 10 8 05/30 Read block 300 len 16384 06/30 Read block 511 len 20480 07/30 Read block 276 len 20480 08/30 Flush 09/30 Write block 425 len 12288 data: 8 9 9 10/30 Read block 521 len 28672 11/30 Write block 237 len 32768 data: 5 9 9 9 12 13 13 13 12/30 Read block 271 len 36864 13/30 Write block 404 len 40960 data: 7 8 9 9 7 6 8 8 7 7 14/30 Read block 72 len 24576 15/30 Write block 246 len 28672 data: 13 11 10 9 10 9 7 16/30 Read block 194 len 36864 17/30 Write block 69 len 32768 data: 7 6 7 7 6 6 6 7 18/30 Write block 170 len 16384 data: 9 10 10 8 19/30 Read block 172 len 36864 20/30 Read block 257 len 8192 21/30 Write block 228 len 24576 data: 15 15 15 12 9 7 22/30 Read block 314 len 28672 23/30 Write block 11 len 28672 data: 12 13 14 14 12 10 10 24/30 Write block 392 len 28672 data: 12 12 9 9 8 8 9 25/30 Write block 459 len 16384 data: 14 12 11 9 26/30 Write block 154 len 8192 data: 4 4 27/30 Write block 139 len 8192 data: 13 17 28/30 Read block 90 len 40960 29/30 Read block 574 len 36864 30/30 Read block 295 len 40960 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 6 Done Done Done false 1004 Acked Read 4 Done Done Sent false 1005 Acked Read 5 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Write 8 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Read 6 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Write 7 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 10 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 26 27 Done 27 26 1 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) Some(JobId(1007)) Some(JobId(1002)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:40.195254049Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1617} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 220 220 220 T T F <--- 1 020-039 116 116 116 217 217 217 F F F 2 040-059 117 117 117 219 219 219 F F F 3 060-079 119 119 119 221 221 221 T T F <--- 4 080-099 120 120 114 223 223 214 F F T <--- 5 100-119 113 113 113 213 213 213 F F F 6 120-139 119 119 119 221 221 221 T T F <--- 7 140-159 104 104 104 191 191 191 T T F <--- 8 160-179 118 118 118 220 220 220 T T F <--- 9 180-199 117 117 117 219 219 219 F F F 10 200-219 114 114 114 214 214 214 F F F 11 220-239 111 111 111 210 210 210 T T F <--- 12 240-259 109 109 109 205 205 205 T T F <--- 13 260-279 120 120 120 222 222 222 F F F 14 280-299 117 117 117 219 219 219 F F F 15 300-319 116 116 116 218 218 218 F F F 16 320-339 114 114 114 214 214 214 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 118 118 118 220 220 220 F F F 19 380-399 119 119 119 221 221 221 T T F <--- 20 400-419 119 119 119 221 221 221 T T F <--- 21 420-439 117 117 117 219 219 219 T T F <--- 22 440-459 117 117 117 219 219 219 T T F <--- 23 460-479 113 113 113 213 213 213 T T F <--- 24 480-499 120 120 120 222 222 222 F F F 25 500-519 117 117 117 219 219 219 F F F 26 520-539 117 117 117 219 219 219 F F F 27 540-559 113 113 113 213 213 213 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 118 118 118 220 220 220 F F F Max gen: 120, Max flush: 223 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 224 224 224 F F F 1 020-039 116 116 116 217 217 217 F F F 2 040-059 117 117 117 219 219 219 F F F 3 060-079 121 121 121 224 224 224 F F F 4 080-099 120 120 120 223 223 223 F F F 5 100-119 113 113 113 213 213 213 F F F 6 120-139 121 121 121 224 224 224 F F F 7 140-159 121 121 121 224 224 224 F F F 8 160-179 121 121 121 224 224 224 F F F 9 180-199 117 117 117 219 219 219 F F F 10 200-219 114 114 114 214 214 214 F F F 11 220-239 121 121 121 224 224 224 F F F 12 240-259 121 121 121 224 224 224 F F F 13 260-279 120 120 120 222 222 222 F F F 14 280-299 117 117 117 219 219 219 F F F 15 300-319 116 116 116 218 218 218 F F F 16 320-339 114 114 114 214 214 214 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 118 118 118 220 220 220 F F F 19 380-399 121 121 121 224 224 224 F F F 20 400-419 121 121 121 224 224 224 F F F 21 420-439 121 121 121 224 224 224 F F F 22 440-459 121 121 121 224 224 224 F F F 23 460-479 121 121 121 224 224 224 F F F 24 480-499 120 120 120 222 222 222 F F F 25 500-519 117 117 117 219 219 219 F F F 26 520-539 117 117 117 219 219 219 F F F 27 540-559 113 113 113 213 213 213 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 118 118 118 220 220 220 F F F Max gen: 121, Max flush: 224 Begin loop 61 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.450473628Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.458658122Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.458982435Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459009211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459017849Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459029258Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625} {"msg":"Crucible 913cba3d-d571-49fb-8b0f-d896869096d1 has session id: 4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459349801Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Upstairs opts: Upstairs UUID: 913cba3d-d571-49fb-8b0f-d896869096d1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459365758Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Crucible stats registered with UUID: 913cba3d-d571-49fb-8b0f-d896869096d1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459375477Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:44.459384426Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459537383Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"io task","client":"1","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459576298Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"io task","client":"0","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459587036Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"io task","client":"2","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.45969691Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"task":"crutest"} {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459711168Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625} {"msg":"913cba3d-d571-49fb-8b0f-d896869096d1 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459745393Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459755122Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"client":"0","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.4597648Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"client":"1","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459776658Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"client":"2","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459786157Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"io task","client":"2","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459796086Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"io task","client":"0","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459805914Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"io task","client":"1","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:63762)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459848328Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"io task","client":"0","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:60425)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459859296Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"io task","client":"1","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"ds_connection connected from Ok(127.0.0.1:54255)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.459879423Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"io task","client":"2","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460544126Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"client":"0","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460568843Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"client":"1","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460649651Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"client":"2","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460829285Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460900044Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"[0]R flush_numbers[0..12]: [224, 217, 219, 224, 223, 213, 224, 224, 224, 219, 214, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460920411Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"[0]R generation[0..12]: [121, 116, 117, 121, 120, 113, 121, 121, 121, 117, 114, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.46093107Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460942068Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"[1]R flush_numbers[0..12]: [224, 217, 219, 224, 223, 213, 224, 224, 224, 219, 214, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460952627Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"[1]R generation[0..12]: [121, 116, 117, 121, 120, 113, 121, 121, 121, 117, 114, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460963115Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460973953Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"[2]R flush_numbers[0..12]: [224, 217, 219, 224, 223, 213, 224, 224, 224, 219, 214, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.460984392Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"[2]R generation[0..12]: [121, 116, 117, 121, 120, 113, 121, 121, 121, 117, 114, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.46099473Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461005569Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461014907Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461024656Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Next flush: 225","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461033955Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461043313Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"":"downstairs","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461052812Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.4610625Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"client":"0","":"downstairs","session_id"Wait for a query_work_queue command to finish before sending IO :"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461076238Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"client":"1","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461085777Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"client":"2","":"downstairs","session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461095516Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461104554Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"913cba3d-d571-49fb-8b0f-d896869096d1 is now active with session: 4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461114533Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461123512Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.46113092Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625,"session_id":"4c7d16c0-374f-49f8-80ba-ae94e6c3f4b9"} {"msg":"Activated sub_volume 913cba3d-d571-49fb-8b0f-d896869096d1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:44.461139869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 538 len 36864 data: 13 11 9 9 9 8 6 6 7 02/30 Read block 235 len 28672 03/30 Read block 544 len 40960 04/30 Read block 141 len 8192 05/30 Read block 320 len 40960 06/30 Flush 07/30 Write block 88 len 12288 data: 13 14 12 08/30 Flush 09/30 Write block 436 len 4096 data: 16 10/30 Write block 61 len 36864 data: 9 9 8 11 12 9 8 7 8 11/30 Read block 394 len 24576 12/30 Read block 98 len 20480 13/30 Write block 100 len 20480 data: 15 14 11 12 9 14/30 Flush 15/30 Write block 175 len 36864 data: 8 11 12 14 13 14 12 13 12 16/30 Read block 291 len 32768 17/30 Write block 315 len 32768 data: 12 15 15 14 13 12 7 6 18/30 Write block 468 len 4096 data: 16 19/30 Read block 390 len 28672 20/30 Write block 84 len 8192 data: 10 12 21/30 Write block 421 len 28672 data: 6 9 8 8 9 10 10 22/30 Flush 23/30 Write block 202 len 16384 data: 6 7 11 9 24/30 Read block 321 len 20480 25/30 Write block 23 len 4096 data: 5 26/30 Write block 378 len 12288 data: 9 11 9 27/30 Write block 459 len 36864 data: 15 13 12 10 10 7 6 9 12 28/30 Write block 587 len 8192 data: 11 10 29/30 Read block 260 len 32768 30/30 Read block 494 len 24576 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 10 Done Sent Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 3 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 1 Done Sent Done false 1009 Acked Write 9 Done Sent Done false 1010 Acked Read 6 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Write 5 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Write 8 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 4 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 0 28 Done 30 2 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:44.470222752Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1625} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 224 224 224 F F F 1 020-039 116 116 116 217 217 217 T F T <--- 2 040-059 117 117 117 219 219 219 F F F 3 060-079 122 121 122 227 224 227 F F F <--- 4 080-099 122 120 122 228 223 228 F F F <--- 5 100-119 122 113 122 227 213 227 F F F <--- 6 120-139 121 121 121 224 224 224 F F F 7 140-159 121 121 121 224 224 224 F F F 8 160-179 122 121 122 228 224 228 F F F <--- 9 180-199 122 117 122 228 219 228 F F F <--- 10 200-219 114 114 114 214 214 214 T F T <--- 11 220-239 121 121 121 224 224 224 F F F 12 240-259 121 121 121 224 224 224 F F F 13 260-279 120 120 120 222 222 222 F F F 14 280-299 117 117 117 219 219 219 F F F 15 300-319 122 116 122 228 218 228 F F F <--- 16 320-339 122 114 122 228 214 228 F F F <--- 17 340-359 118 118 118 220 220 220 F F F 18 360-379 118 118 118 220 220 220 T F T <--- 19 380-399 121 121 121 224 224 224 T F T <--- 20 400-419 121 121 121 224 224 224 F F F 21 420-439 122 121 122 228 224 228 F F F <--- 22 440-459 121 121 121 224 224 224 T F T <--- 23 460-479 122 121 122 228 224 228 T F T <--- 24 480-499 120 120 120 222 222 222 F F F 25 500-519 117 117 117 219 219 219 F F F 26 520-539 122 117 122 225 219 225 F T F <--- 27 540-559 122 113 122 225 213 225 F T F <--- 28 560-579 118 118 118 220 220 220 F F F 29 580-599 118 118 118 220 220 220 T F T <--- Max gen: 122, Max flush: 228 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 224 224 224 F F F 1 020-039 123 123 123 229 229 229 F F F 2 040-059 117 117 117 219 219 219 F F F 3 060-079 122 122 122 227 227 227 F F F 4 080-099 122 122 122 228 228 228 F F F 5 100-119 122 122 122 227 227 227 F F F 6 120-139 121 121 121 224 224 224 F F F 7 140-159 121 121 121 224 224 224 F F F 8 160-179 122 122 122 228 228 228 F F F 9 180-199 122 122 122 228 228 228 F F F 10 200-219 123 123 123 229 229 229 F F F 11 220-239 121 121 121 224 224 224 F F F 12 240-259 121 121 121 224 224 224 F F F 13 260-279 120 120 120 222 222 222 F F F 14 280-299 117 117 117 219 219 219 F F F 15 300-319 122 122 122 228 228 228 F F F 16 320-339 122 122 122 228 228 228 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 123 123 123 229 229 229 F F F 19 380-399 123 123 123 229 229 229 F F F 20 400-419 121 121 121 224 224 224 F F F 21 420-439 122 122 122 228 228 228 F F F 22 440-459 123 123 123 229 229 229 F F F 23 460-479 123 123 123 229 229 229 F F F 24 480-499 120 120 120 222 222 222 F F F 25 500-519 117 117 117 219 219 219 F F F 26 520-539 122 122 122 225 225 225 F F F 27 540-559 122 122 122 225 225 225 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 123, Max flush: 229 Begin loop 62 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.734841466Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:48.742031106Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.742086408Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749067087Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749325729Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749354335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749363114Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749377362Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633} {"msg":"Crucible 541530ea-2d17-4cf8-879e-b4784aade09b has session id: 10468ab9-4eba-4332-9bb8-f06512dc4110","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749737799Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Upstairs opts: Upstairs UUID: 541530ea-2d17-4cf8-879e-b4784aade09b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749758676Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Crucible stats registered with UUID: 541530ea-2d17-4cf8-879e-b4784aade09b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749768415Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:48.749777333Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749947908Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"io task","client":"0","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749986243Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"io task","client":"2","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.749997271Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"io task","client":"1","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750125243Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"task":"crutest"} {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750139041Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633} {"msg":"541530ea-2d17-4cf8-879e-b4784aade09b active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750162217Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750171536Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"client":"0","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750181394Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"client":"1","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750191023Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"client":"2","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750200662Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"io task","client":"2","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.75021277Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"io task","client":"0","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750223258Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"io task","client":"1","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"ds_connection connected from Ok(127.0.0.1:51841)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750272211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"io task","client":"0","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"ds_connection connected from Ok(127.0.0.1:64495)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750297477Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"io task","client":"2","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"ds_connection connected from Ok(127.0.0.1:43573)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.75035132Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"io task","client":"1","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750921796Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"client":"1","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.750943523Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"client":"2","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.75103509Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"client":"0","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.75123911Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751273615Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"[0]R flush_numbers[0..12]: [224, 229, 219, 227, 228, 227, 224, 224, 228, 228, 229, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751326837Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"[0]R generation[0..12]: [121, 123, 117, 122, 122, 122, 121, 121, 122, 122, 123, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751338125Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751349034Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"[1]R flush_numbers[0..12]: [224, 229, 219, 227, 228, 227, 224, 224, 228, 228, 229, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751359652Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"[1]R generation[0..12]: [121, 123, 117, 122, 122, 122, 121, 121, 122, 122, 123, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.75137223Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751383069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"[2]R flush_numbers[0..12]: [224, 229, 219, 227, 228, 227, 224, 224, 228, 228, 229, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751393367Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"[2]R generation[0..12]: [121, 123, 117, 122, 122, 122, 121, 121, 122, 122, 123, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751403986Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751414924Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751424243Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751433771Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Next flush: 230","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.75144305Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751452269Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"":"downstairs","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751462197Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751470996Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"client":"0","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751481064Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633Wait for a query_work_queue command to finish before sending IO ,"client":"1","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751494812Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"client":"2","":"downstairs","session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751504831Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.7515139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"541530ea-2d17-4cf8-879e-b4784aade09b is now active with session: 10468ab9-4eba-4332-9bb8-f06512dc4110","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751527348Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751536316Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751543745Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633,"session_id":"10468ab9-4eba-4332-9bb8-f06512dc4110"} {"msg":"Activated sub_volume 541530ea-2d17-4cf8-879e-b4784aade09b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:48.751552844Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 466 len 40960 data: 10 13 17 14 14 15 14 15 14 11 02/30 Write block 58 len 20480 data: 10 9 8 10 10 03/30 Read block 447 len 8192 04/30 Read block 240 len 24576 05/30 Read block 503 len 28672 06/30 Write block 35 len 36864 data: 5 5 7 8 9 10 11 12 12 07/30 Flush 08/30 Write block 81 len 36864 data: 9 10 9 11 13 11 14 14 15 09/30 Write block 328 len 12288 data: 12 10 13 10/30 Read block 61 len 24576 11/30 Write block 194 len 20480 data: 12 12 11 12 12 12/30 Write block 215 len 12288 data: 4 5 6 13/30 Write block 274 len 40960 data: 10 6 6 7 5 11 10 11 8 10 14/30 Write block 37 len 12288 data: 8 9 10 15/30 Write block 361 len 4096 data: 6 16/30 Read block 205 len 4096 17/30 Flush 18/30 Write block 428 len 40960 data: 7 7 9 11 12 15 17 18 17 15 19/30 Flush 20/30 Write block 262 len 16384 data: 10 12 13 13 21/30 Flush 22/30 Write block 16 len 12288 data: 11 11 10 23/30 Read block 93 len 20480 24/30 Write block 183 len 8192 data: 13 10 25/30 Read block 15 len 16384 26/30 Write block 539 len 8192 data: 12 10 27/30 Write block 419 len 20480 data: 7 7 7 10 9 28/30 Read block 442 len 28672 29/30 Write block 101 len 36864 data: 15 12 13 10 10 10 11 11 12 30/30 Write block 69 len 32768 data: 9 7 8 8 7 7 7 8 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 5 Sent Done Done false 1002 Acked Read 2 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 7 Sent Done Done false 1005 Acked Write 9 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Write 3 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Write 3 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Read 1 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 5 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 2 2 33 Done 1 28 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:48.760452423Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1633} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 224 224 224 F T T <--- 1 020-039 123 124 124 229 231 231 F F F <--- 2 040-059 117 124 124 219 230 230 F F F <--- 3 060-079 122 124 124 227 230 230 F T T <--- 4 080-099 122 124 124 228 231 231 F F F <--- 5 100-119 122 122 122 227 227 227 F T T <--- 6 120-139 121 121 121 224 224 224 F F F 7 140-159 121 121 121 224 224 224 F F F 8 160-179 122 122 122 228 228 228 F F F 9 180-199 122 124 124 228 231 231 F T T <--- 10 200-219 123 124 124 229 231 231 F F F <--- 11 220-239 121 121 121 224 224 224 F F F 12 240-259 121 121 121 224 224 224 F F F 13 260-279 120 124 124 222 233 233 F F F <--- 14 280-299 117 124 124 219 231 231 F F F <--- 15 300-319 122 122 122 228 228 228 F F F 16 320-339 122 124 124 228 231 231 F F F <--- 17 340-359 118 118 118 220 220 220 F F F 18 360-379 123 124 124 229 231 231 F F F <--- 19 380-399 123 123 123 229 229 229 F F F 20 400-419 121 121 121 224 224 224 F T T <--- 21 420-439 122 124 124 228 232 232 F T T <--- 22 440-459 123 123 123 229 229 229 F F F 23 460-479 123 124 124 229 230 230 T F F <--- 24 480-499 120 120 120 222 222 222 F F F 25 500-519 117 117 117 219 219 219 F F F 26 520-539 122 122 122 225 225 225 F T T <--- 27 540-559 122 122 122 225 225 225 F T T <--- 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 124, Max flush: 233 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 234 234 234 F F F 1 020-039 124 124 124 231 231 231 F F F 2 040-059 124 124 124 230 230 230 F F F 3 060-079 125 125 125 234 234 234 F F F 4 080-099 124 124 124 231 231 231 F F F 5 100-119 125 125 125 234 234 234 F F F 6 120-139 121 121 121 224 224 224 F F F 7 140-159 121 121 121 224 224 224 F F F 8 160-179 122 122 122 228 228 228 F F F 9 180-199 125 125 125 234 234 234 F F F 10 200-219 124 124 124 231 231 231 F F F 11 220-239 121 121 121 224 224 224 F F F 12 240-259 121 121 121 224 224 224 F F F 13 260-279 124 124 124 233 233 233 F F F 14 280-299 124 124 124 231 231 231 F F F 15 300-319 122 122 122 228 228 228 F F F 16 320-339 124 124 124 231 231 231 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 124 124 124 231 231 231 F F F 19 380-399 123 123 123 229 229 229 F F F 20 400-419 125 125 125 234 234 234 F F F 21 420-439 125 125 125 234 234 234 F F F 22 440-459 123 123 123 229 229 229 F F F 23 460-479 124 124 124 230 230 230 F F F 24 480-499 120 120 120 222 222 222 F F F 25 500-519 117 117 117 219 219 219 F F F 26 520-539 125 125 125 234 234 234 F F F 27 540-559 125 125 125 234 234 234 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 125, Max flush: 234 Begin loop 63 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.030067896Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:53.037316267Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.037380337Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.044224897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.044497377Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.044525213Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.044533962Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.04454534Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641} {"msg":"Crucible 084663f4-9a59-4cb9-b7d0-97f342ba7a21 has session id: faffb79c-4bde-48dd-955b-f1393a5db02e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.044870253Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Upstairs opts: Upstairs UUID: 084663f4-9a59-4cb9-b7d0-97f342ba7a21, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.044894539Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Crucible stats registered with UUID: 084663f4-9a59-4cb9-b7d0-97f342ba7a21","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.044906317Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:53.044915306Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045088601Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"io task","client":"2","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045128835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"io task","client":"0","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045139823Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"io task","client":"1","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.04522938Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"task":"crutest"} {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045247448Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641} {"msg":"084663f4-9a59-4cb9-b7d0-97f342ba7a21 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045284882Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045294271Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"client":"0","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045303869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"client":"1","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045313478Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"client":"2","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045323087Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"io task","client":"2","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045332925Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"io task","client":"1","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045378558Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"io task","client":"0","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"ds_connection connected from Ok(127.0.0.1:58401)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045393216Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"io task","client":"2","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"ds_connection connected from Ok(127.0.0.1:48547)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045403465Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"io task","client":"1","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"ds_connection connected from Ok(127.0.0.1:34030)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.045537715Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"io task","client":"0","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046206097Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"client":"2","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046229094Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"client":"0","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046240482Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"client":"1","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046520372Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046549027Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"[0]R flush_numbers[0..12]: [234, 231, 230, 234, 231, 234, 224, 224, 228, 234, 231, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046610918Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"[0]R generation[0..12]: [125, 124, 124, 125, 124, 125, 121, 121, 122, 125, 124, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046622247Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046633345Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"[1]R flush_numbers[0..12]: [234, 231, 230, 234, 231, 234, 224, 224, 228, 234, 231, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046644053Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"[1]R generation[0..12]: [125, 124, 124, 125, 124, 125, 121, 121, 122, 125, 124, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046654482Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.04666533Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"[2]R flush_numbers[0..12]: [234, 231, 230, 234, 231, 234, 224, 224, 228, 234, 231, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046676239Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"[2]R generation[0..12]: [125, 124, 124, 125, 124, 125, 121, 121, 122, 125, 124, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046686707Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046699625Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046709074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046718843Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Next flush: 235","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046728111Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.04673746Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"":"downstairs","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046747158Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046755837Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"client":"0","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046765546Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"client":"1","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046775154Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"client":"2","":"downstairs","session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046785023Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046794072Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"084663f4-9a59-4cb9-b7d0-97f342ba7a21 is now active with session: faffb79c-4bde-48dd-955b-f1393a5db02e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.04680408Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046813269Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641} {"msg":"Wait for a query_work_queue command to finish before sending IO Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046823987Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641,"session_id":"faffb79c-4bde-48dd-955b-f1393a5db02e"} {"msg":"Activated sub_volume 084663f4-9a59-4cb9-b7d0-97f342ba7a21","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:53.046833296Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 337 len 12288 data: 8 9 10 03/30 Read block 184 len 4096 04/30 Write block 399 len 20480 data: 7 7 6 7 6 05/30 Write block 496 len 12288 data: 7 10 9 06/30 Read block 269 len 28672 07/30 Write block 127 len 12288 data: 9 8 9 08/30 Write block 427 len 12288 data: 11 8 8 09/30 Write block 293 len 28672 data: 11 9 6 6 6 7 6 10/30 Write block 456 len 20480 data: 15 15 17 16 14 11/30 Flush 12/30 Write block 398 len 36864 data: 10 8 8 7 8 7 8 9 10 13/30 Read block 486 len 40960 14/30 Write block 399 len 36864 data: 9 9 8 9 8 9 10 11 10 15/30 Read block 78 len 32768 16/30 Read block 481 len 36864 17/30 Read block 494 len 32768 18/30 Read block 131 len 20480 19/30 Read block 395 len 16384 20/30 Read block 292 len 32768 21/30 Read block 499 len 8192 22/30 Write block 106 len 8192 data: 11 12 23/30 Write block 432 len 36864 data: 13 16 18 19 18 16 18 13 12 24/30 Write block 401 len 20480 data: 9 10 9 10 11 25/30 Write block 187 len 36864 data: 8 7 7 10 11 13 13 13 13 26/30 Write block 425 len 32768 data: 10 11 12 9 9 10 12 14 27/30 Read block 204 len 24576 28/30 Read block 70 len 16384 29/30 Write block 67 len 36864 data: 9 8 10 8 9 9 8 8 8 30/30 Read block 287 len 16384 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Write 3 Done Done Done false 1005 Acked Read 7 Done Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Write 7 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Read 8 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Write 9 Sent Done Done false 1029 Acked Read 4 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 0 0 24 Done 5 29 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:53.056347025Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1641} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 234 234 234 F F F 1 020-039 124 124 124 231 231 231 F F F 2 040-059 124 124 124 230 230 230 F F F 3 060-079 125 125 125 234 234 234 F T T <--- 4 080-099 124 124 124 231 231 231 F F F 5 100-119 125 125 125 234 234 234 F T T <--- 6 120-139 121 126 126 224 236 236 F F F <--- 7 140-159 121 121 121 224 224 224 F F F 8 160-179 122 122 122 228 228 228 F F F 9 180-199 125 125 125 234 234 234 F T T <--- 10 200-219 124 124 124 231 231 231 F F F 11 220-239 121 121 121 224 224 224 F F F 12 240-259 121 121 121 224 224 224 F F F 13 260-279 124 124 124 233 233 233 F F F 14 280-299 124 126 126 231 236 236 F F F <--- 15 300-319 122 122 122 228 228 228 F F F 16 320-339 124 126 126 231 236 236 T F F <--- 17 340-359 118 118 118 220 220 220 F F F 18 360-379 124 124 124 231 231 231 F F F 19 380-399 123 126 126 229 236 236 T T T <--- 20 400-419 125 126 126 234 236 236 T T T <--- 21 420-439 125 126 126 234 236 236 F T T <--- 22 440-459 123 126 126 229 236 236 F T T <--- 23 460-479 124 126 126 230 236 236 F F F <--- 24 480-499 120 126 126 222 236 236 T F F <--- 25 500-519 117 117 117 219 219 219 F F F 26 520-539 125 125 125 234 234 234 F F F 27 540-559 125 125 125 234 234 234 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 126, Max flush: 236 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 234 234 234 F F F 1 020-039 124 124 124 231 231 231 F F F 2 040-059 124 124 124 230 230 230 F F F 3 060-079 127 127 127 237 237 237 F F F 4 080-099 124 124 124 231 231 231 F F F 5 100-119 127 127 127 237 237 237 F F F 6 120-139 126 126 126 236 236 236 F F F 7 140-159 121 121 121 224 224 224 F F F 8 160-179 122 122 122 228 228 228 F F F 9 180-199 127 127 127 237 237 237 F F F 10 200-219 124 124 124 231 231 231 F F F 11 220-239 121 121 121 224 224 224 F F F 12 240-259 121 121 121 224 224 224 F F F 13 260-279 124 124 124 233 233 233 F F F 14 280-299 126 126 126 236 236 236 F F F 15 300-319 122 122 122 228 228 228 F F F 16 320-339 126 126 126 236 236 236 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 124 124 124 231 231 231 F F F 19 380-399 127 127 127 237 237 237 F F F 20 400-419 127 127 127 237 237 237 F F F 21 420-439 127 127 127 237 237 237 F F F 22 440-459 127 127 127 237 237 237 F F F 23 460-479 126 126 126 236 236 236 F F F 24 480-499 126 126 126 236 236 236 F F F 25 500-519 117 117 117 219 219 219 F F F 26 520-539 125 125 125 234 234 234 F F F 27 540-559 125 125 125 234 234 234 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 127, Max flush: 237 Begin loop 64 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.316076615Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.324432843Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.324726471Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.324753837Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.324762275Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.324773524Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649} {"msg":"Crucible 6443eb24-d4c3-4234-a87d-cc9811e14e60 has session id: c5688ac4-d1f2-47d1-88db-2fcb8bda3256","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325160987Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Upstairs opts: Upstairs UUID: 6443eb24-d4c3-4234-a87d-cc9811e14e60, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325190773Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Crucible stats registered with UUID: 6443eb24-d4c3-4234-a87d-cc9811e14e60","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325202181Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:57.32521146Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325356578Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"io task","client":"1","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325379005Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"io task","client":"0","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325389054Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"io task","client":"2","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325541111Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"task":"crutest"} {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325564258Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649} {"msg":"6443eb24-d4c3-4234-a87d-cc9811e14e60 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325573107Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325584315Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"client":"0","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325594724Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"client":"1","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325604352Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"client":"2","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325613841Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"io task","client":"2","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325623859Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"io task","client":"0","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325654225Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"io task","client":"1","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"ds_connection connected from Ok(127.0.0.1:39215)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325675352Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"io task","client":"0","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"ds_connection connected from Ok(127.0.0.1:45067)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325692229Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"io task","client":"2","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"ds_connection connected from Ok(127.0.0.1:46414)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.325804573Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"io task","client":"1","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326368151Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"client":"2","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326403895Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"client":"0","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326497482Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"client":"1","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326699372Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326718239Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"[0]R flush_numbers[0..12]: [234, 231, 230, 237, 231, 237, 236, 224, 228, 237, 231, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.32678606Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"[0]R generation[0..12]: [125, 124, 124, 127, 124, 127, 126, 121, 122, 127, 124, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326800367Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326811636Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"[1]R flush_numbers[0..12]: [234, 231, 230, 237, 231, 237, 236, 224, 228, 237, 231, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326822174Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"[1]R generation[0..12]: [125, 124, 124, 127, 124, 127, 126, 121, 122, 127, 124, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326832653Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326843671Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"[2]R flush_numbers[0..12]: [234, 231, 230, 237, 231, 237, 236, 224, 228, 237, 231, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.32685423Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"[2]R generation[0..12]: [125, 124, 124, 127, 124, 127, 126, 121, 122, 127, 124, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326864658Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326875496Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Max found gen is 128","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326884815Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326894464Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Next flush: 238","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326903772Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326913171Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"":"downstairs","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.32692276Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326931448Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"client":"0","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326942557Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"client":"1","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326952635Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"client":"2","":"downstairs","session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326962724Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326971952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"6443eb24-d4c3-4234-a87d-cc9811e14e60 is now active with session: c5688ac4-d1f2-47d1-88db-2fcb8bda3256","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:14:57.326981751Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.326992999Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.327011597Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649,"session_id":"c5688ac4-d1f2-47d1-88db-2fcb8bda3256"} {"msg":"Activated sub_volume 6443eb24-d4c3-4234-a87d-cc9811e14e60","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:14:57.327020755Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 157 len 40960 data: 7 8 10 9 8 9 17 18 17 14 02/30 Write block 483 len 36864 data: 9 7 7 6 6 5 5 6 6 03/30 Flush 04/30 Flush 05/30 Write block 40 len 8192 data: 11 12 06/30 Read block 196 len 32768 07/30 Read block 364 len 36864 08/30 Read block 458 len 20480 09/30 Read block 511 len 32768 10/30 Flush 11/30 Write block 200 len 32768 data: 9 8 7 8 12 10 8 7 12/30 Write block 539 len 24576 data: 13 11 10 10 9 7 13/30 Read block 318 len 16384 14/30 Read block 233 len 4096 15/30 Read block 475 len 24576 16/30 Read block 457 len 36864 17/30 Read block 248 len 36864 18/30 Read block 110 len 24576 19/30 Read block 490 len 24576 20/30 Read block 226 len 24576 21/30 Write block 285 len 36864 data: 10 9 7 9 8 9 9 10 12 22/30 Write block 206 len 20480 data: 9 8 5 5 4 23/30 Read block 534 len 36864 24/30 Flush 25/30 Write block 193 len 16384 data: 14 14 14 12 26/30 Read block 589 len 32768 27/30 Write block 505 len 4096 data: 6 28/30 Write block 285 len 28672 data: 11 10 8 10 9 10 10 29/30 Write block 22 len 4096 data: 5 30/30 Write block 375 len 32768 data: 9 12 13 10 12 10 7 8 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 2 Done Done Done false 1005 Acked Read 8 Done Done Done false 1006 Acked Read 9 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Write 9 Done Sent Done false 1021 Acked Write 5 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Write 7 Sent Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 24 3 30 Done 23 2 23 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1003)) Some(JobId(1023)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:14:57.336241717Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1649} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 234 234 234 F F F 1 020-039 124 124 124 231 231 231 T F T <--- 2 040-059 128 124 128 240 230 240 F T F <--- 3 060-079 127 127 127 237 237 237 F F F 4 080-099 124 124 124 231 231 231 F F F 5 100-119 127 127 127 237 237 237 F F F 6 120-139 126 126 126 236 236 236 F F F 7 140-159 128 128 128 238 238 238 F F F 8 160-179 128 128 128 238 238 238 F F F 9 180-199 127 127 127 237 237 237 T F T <--- 10 200-219 128 124 128 241 231 241 F F F <--- 11 220-239 121 121 121 224 224 224 F F F 12 240-259 121 121 121 224 224 224 F F F 13 260-279 124 124 124 233 233 233 F F F 14 280-299 128 126 128 241 236 241 T F T <--- 15 300-319 122 122 122 228 228 228 F F F 16 320-339 126 126 126 236 236 236 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 124 124 124 231 231 231 T F T <--- 19 380-399 127 127 127 237 237 237 T F T <--- 20 400-419 127 127 127 237 237 237 F F F 21 420-439 127 127 127 237 237 237 F F F 22 440-459 127 127 127 237 237 237 F F F 23 460-479 126 126 126 236 236 236 F F F 24 480-499 128 128 128 238 238 238 F F F 25 500-519 117 117 117 219 219 219 T F T <--- 26 520-539 128 125 128 241 234 241 F F F <--- 27 540-559 128 125 128 241 234 241 F F F <--- 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 128, Max flush: 241 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 234 234 234 F F F 1 020-039 129 129 129 242 242 242 F F F 2 040-059 128 128 128 240 240 240 F F F 3 060-079 127 127 127 237 237 237 F F F 4 080-099 124 124 124 231 231 231 F F F 5 100-119 127 127 127 237 237 237 F F F 6 120-139 126 126 126 236 236 236 F F F 7 140-159 128 128 128 238 238 238 F F F 8 160-179 128 128 128 238 238 238 F F F 9 180-199 129 129 129 242 242 242 F F F 10 200-219 128 128 128 241 241 241 F F F 11 220-239 121 121 121 224 224 224 F F F 12 240-259 121 121 121 224 224 224 F F F 13 260-279 124 124 124 233 233 233 F F F 14 280-299 129 129 129 242 242 242 F F F 15 300-319 122 122 122 228 228 228 F F F 16 320-339 126 126 126 236 236 236 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 129 129 129 242 242 242 F F F 19 380-399 129 129 129 242 242 242 F F F 20 400-419 127 127 127 237 237 237 F F F 21 420-439 127 127 127 237 237 237 F F F 22 440-459 127 127 127 237 237 237 F F F 23 460-479 126 126 126 236 236 236 F F F 24 480-499 128 128 128 238 238 238 F F F 25 500-519 129 129 129 242 242 242 F F F 26 520-539 128 128 128 241 241 241 F F F 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 129, Max flush: 242 Begin loop 65 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.582434725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:01.58958959Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.5896524Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.596746604Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597003706Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597029482Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597037891Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597049309Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657} {"msg":"Crucible d4a1efad-785f-45bc-b1ae-dc0e92644cb9 has session id: a895d5a1-9595-48c6-8420-0cb0bea3209d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597347516Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Upstairs opts: Upstairs UUID: d4a1efad-785f-45bc-b1ae-dc0e92644cb9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597368133Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Crucible stats registered with UUID: d4a1efad-785f-45bc-b1ae-dc0e92644cb9","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597377961Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:01.59738683Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597564804Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"io task","client":"1","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597587621Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"io task","client":"0","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597597599Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"io task","client":"2","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597689636Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"task":"crutest"} {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597702944Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657} {"msg":"d4a1efad-785f-45bc-b1ae-dc0e92644cb9 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597734489Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597743728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"client":"0","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597753326Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"client":"1","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597762825Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"client":"2","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597772254Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"io task","client":"2","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597787032Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"io task","client":"1","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"ds_connection connected from Ok(127.0.0.1:42275)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.59779825Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"io task","client":"2","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597834095Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"io task","client":"0","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"ds_connection connected from Ok(127.0.0.1:55363)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.597858711Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"io task","client":"1","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"ds_connection connected from Ok(127.0.0.1:58218)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598020667Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"io task","client":"0","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598494038Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"client":"1","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598520314Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"client":"0","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598557899Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"client":"2","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598809572Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598841997Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"[0]R flush_numbers[0..12]: [234, 242, 240, 237, 231, 237, 236, 238, 238, 242, 241, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.59889491Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"[0]R generation[0..12]: [125, 129, 128, 127, 124, 127, 126, 128, 128, 129, 128, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598906368Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598917396Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"[1]R flush_numbers[0..12]: [234, 242, 240, 237, 231, 237, 236, 238, 238, 242, 241, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598927945Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"[1]R generation[0..12]: [125, 129, 128, 127, 124, 127, 126, 128, 128, 129, 128, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598938403Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598949222Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"[2]R flush_numbers[0..12]: [234, 242, 240, 237, 231, 237, 236, 238, 238, 242, 241, 224]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.59896157Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"[2]R generation[0..12]: [125, 129, 128, 127, 124, 127, 126, 128, 128, 129, 128, 121]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598972108Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598982887Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.598992295Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599002004Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Next flush: 243","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599011443Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599020771Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"":"downstairs","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.59903037Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599039189Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"client":"0","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599048907Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"client":"1","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599058496Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"client":"2","":"downstairs","session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599068434Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599077503Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"d4a1efad-785f-45bc-b1ae-dc0e92644cb9 is now active with session: a895d5a1-9595-48c6-8420-0cb0bea3209d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599087471Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:15:01.59909647Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599106529Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657,"session_id":"a895d5a1-9595-48c6-8420-0cb0bea3209d"} {"msg":"Activated sub_volume d4a1efad-785f-45bc-b1ae-dc0e92644cb9","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:01.599117067Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Flush 03/30 Flush 04/30 Write block 207 len 12288 data: 9 6 6 05/30 Write block 20 len 4096 data: 8 06/30 Read block 335 len 8192 07/30 Write block 164 len 20480 data: 19 18 15 14 11 08/30 Write block 533 len 8192 data: 15 14 09/30 Write block 310 len 4096 data: 11 10/30 Read block 192 len 8192 11/30 Read block 328 len 4096 12/30 Read block 514 len 32768 13/30 Write block 361 len 12288 data: 7 3 4 14/30 Write block 173 len 32768 data: 9 9 9 12 13 15 14 15 15/30 Read block 182 len 40960 16/30 Flush 17/30 Write block 486 len 32768 data: 7 7 6 6 7 7 5 5 18/30 Flush 19/30 Write block 2 len 12288 data: 7 7 9 20/30 Read block 377 len 12288 21/30 Read block 587 len 24576 22/30 Write block 223 len 4096 data: 12 23/30 Read block 182 len 24576 24/30 Read block 230 len 8192 25/30 Write block 37 len 28672 data: 9 10 11 12 13 13 13 26/30 Flush 27/30 Write block 497 len 36864 data: 11 10 10 10 8 6 4 3 7 28/30 Read block 320 len 4096 29/30 Read block 510 len 8192 30/30 Write block 470 len 24576 data: 15 16 15 16 15 12 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 3 Done Done Done false 1004 Acked Write 1 Done Done Done false 1005 Acked Read 2 Done Done Done false 1006 Acked Write 5 Done Done Done false 1007 Acked Write 2 Done Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 22 1 1 24 Done 5 26 26 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:01.608304804Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1657} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 130 130 234 248 248 F F F <--- 1 020-039 129 130 130 242 248 248 T F F <--- 2 040-059 128 130 130 240 248 248 F F F <--- 3 060-079 127 127 127 237 237 237 F F F 4 080-099 124 124 124 231 231 231 F F F 5 100-119 127 127 127 237 237 237 F F F 6 120-139 126 126 126 236 236 236 F F F 7 140-159 128 128 128 238 238 238 F F F 8 160-179 128 130 130 238 246 246 T F F <--- 9 180-199 129 130 130 242 246 246 F F F <--- 10 200-219 128 130 130 241 246 246 T F F <--- 11 220-239 121 130 130 224 248 248 F F F <--- 12 240-259 121 121 121 224 224 224 F F F 13 260-279 124 124 124 233 233 233 F F F 14 280-299 129 129 129 242 242 242 F F F 15 300-319 122 130 130 228 246 246 F F F <--- 16 320-339 126 126 126 236 236 236 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 129 130 130 242 246 246 F F F <--- 19 380-399 129 129 129 242 242 242 F F F 20 400-419 127 127 127 237 237 237 F F F 21 420-439 127 127 127 237 237 237 F F F 22 440-459 127 127 127 237 237 237 F F F 23 460-479 126 126 126 236 236 236 F T T <--- 24 480-499 128 130 130 238 247 247 F T T <--- 25 500-519 129 129 129 242 242 242 F T T <--- 26 520-539 128 130 130 241 246 246 T F F <--- 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 130, Max flush: 248 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 130 130 130 248 248 248 F F F 1 020-039 130 130 130 248 248 248 F F F 2 040-059 130 130 130 248 248 248 F F F 3 060-079 127 127 127 237 237 237 F F F 4 080-099 124 124 124 231 231 231 F F F 5 100-119 127 127 127 237 237 237 F F F 6 120-139 126 126 126 236 236 236 F F F 7 140-159 128 128 128 238 238 238 F F F 8 160-179 130 130 130 246 246 246 F F F 9 180-199 130 130 130 246 246 246 F F F 10 200-219 130 130 130 246 246 246 F F F 11 220-239 130 130 130 248 248 248 F F F 12 240-259 121 121 121 224 224 224 F F F 13 260-279 124 124 124 233 233 233 F F F 14 280-299 129 129 129 242 242 242 F F F 15 300-319 130 130 130 246 246 246 F F F 16 320-339 126 126 126 236 236 236 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 130 130 130 246 246 246 F F F 19 380-399 129 129 129 242 242 242 F F F 20 400-419 127 127 127 237 237 237 F F F 21 420-439 127 127 127 237 237 237 F F F 22 440-459 127 127 127 237 237 237 F F F 23 460-479 131 131 131 249 249 249 F F F 24 480-499 131 131 131 249 249 249 F F F 25 500-519 131 131 131 249 249 249 F F F 26 520-539 130 130 130 246 246 246 F F F 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 131, Max flush: 249 Begin loop 66 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.869295759Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.877802716Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.87811211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878138127Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878146875Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878158354Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665} {"msg":"Crucible c0df249a-d9b3-444f-b007-310a02535080 has session id: 65aa3cea-de9a-4bb4-b771-badf43a89a9e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878509552Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Upstairs opts: Upstairs UUID: c0df249a-d9b3-444f-b007-310a02535080, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878538578Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Crucible stats registered with UUID: c0df249a-d9b3-444f-b007-310a02535080","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878549606Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:05.878558965Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878699954Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"io task","client":"2","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878737679Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"io task","client":"0","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878748637Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"io task","client":"1","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878855492Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"task":"crutest"} {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878876589Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665} {"msg":"c0df249a-d9b3-444f-b007-310a02535080 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878900975Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878910314Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"client":"0","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878919952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"client":"1","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878929811Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"client":"2","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878939289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"io task","client":"2","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878949138Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"io task","client":"0","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.878989572Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"io task","client":"1","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"ds_connection connected from Ok(127.0.0.1:61221)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.8790058Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"io task","client":"0","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"ds_connection connected from Ok(127.0.0.1:59557)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.87907235Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"io task","client":"2","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"ds_connection connected from Ok(127.0.0.1:34498)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.879117453Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"io task","client":"1","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.879691709Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"client":"0","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.879705947Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"client":"1","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.879760169Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"client":"2","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880011293Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880021651Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"[0]R flush_numbers[0..12]: [248, 248, 248, 237, 231, 237, 236, 238, 246, 246, 246, 248]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880058896Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"[0]R generation[0..12]: [130, 130, 130, 127, 124, 127, 126, 128, 130, 130, 130, 130]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880070274Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880081153Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"[1]R flush_numbers[0..12]: [248, 248, 248, 237, 231, 237, 236, 238, 246, 246, 246, 248]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880091811Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"[1]R generation[0..12]: [130, 130, 130, 127, 124, 127, 126, 128, 130, 130, 130, 130]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880102289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880112988Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"[2]R flush_numbers[0..12]: [248, 248, 248, 237, 231, 237, 236, 238, 246, 246, 246, 248]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880123466Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"[2]R generation[0..12]: [130, 130, 130, 127, 124, 127, 126, 128, 130, 130, 130, 130]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880134105Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880144973Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880154462Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.88016416Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Next flush: 250","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880173629Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880182978Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"":"downstairs","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880192616Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880202785Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"client":"0","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880212253Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"client":"1","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880221772Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"client":"2","":"downstairs","session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880231511Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880240599Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"c0df249a-d9b3-444f-b007-310a02535080 is now active with session: 65aa3cea-de9a-4bb4-b771-badf43a89a9e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880254907Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880263936Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880271495Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665,"session_id":"65aa3cea-de9a-4bb4-b771-badf43a89a9e"} {"msg":"Activated sub_volume c0df249a-d9b3-444f-b007-310a02535080","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:05.880280573Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 71 len 24576 data: 10 10 9 9 9 9 02/30 Read block 253 len 40960 03/30 Write block 49 len 36864 data: 13 10 9 9 8 9 9 12 11 04/30 Read block 311 len 40960 05/30 Read block 133 len 24576 06/30 Write block 464 len 28672 data: 8 7 11 14 18 15 16 07/30 Read block 64 len 20480 08/30 Flush 09/30 Write block 35 len 40960 data: 6 6 10 11 12 13 14 14 14 11 10/30 Read block 332 len 12288 11/30 Write block 65 len 40960 data: 13 10 10 9 11 9 11 11 10 10 12/30 Write block 256 len 12288 data: 5 4 4 13/30 Read block 218 len 24576 14/30 Write block 296 len 4096 data: 7 15/30 Read block 167 len 24576 16/30 Read block 526 len 8192 17/30 Write block 117 len 20480 data: 12 10 9 11 11 18/30 Read block 110 len 8192 19/30 Write block 423 len 16384 data: 10 9 11 12 20/30 Write block 176 len 20480 data: 13 14 16 15 16 21/30 Read block 295 len 32768 22/30 Read block 549 len 36864 23/30 Write block 5 len 12288 data: 9 9 9 24/30 Write block 292 len 16384 data: 11 13 10 7 25/30 Read block 282 len 20480 26/30 Write block 26 len 32768 data: 8 6 9 8 8 6 6 5 27/30 Read block 529 len 36864 28/30 Write block 509 len 36864 data: 11 10 9 9 8 11 9 7 9 29/30 Write block 194 len 36864 data: 15 15 13 13 13 11 10 9 8 30/30 Read block 394 len 20480 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 10 Done Done Sent false 1002 Acked Write 9 Done Done Sent false 1003 Acked Read 10 Done Done Sent false 1004 Acked Read 6 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 10 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Read 6 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 5 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Read 5 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Write 9 Done Done Sent false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) Some(JobId(1007)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:05.889703316Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1665} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 130 130 130 248 248 248 T T F <--- 1 020-039 130 130 130 248 248 248 T T F <--- 2 040-059 132 132 130 250 250 248 T T F <--- 3 060-079 132 132 127 250 250 237 T T T <--- 4 080-099 124 124 124 231 231 231 F F F 5 100-119 127 127 127 237 237 237 T T F <--- 6 120-139 126 126 126 236 236 236 T T F <--- 7 140-159 128 128 128 238 238 238 F F F 8 160-179 130 130 130 246 246 246 T T F <--- 9 180-199 130 130 130 246 246 246 T T F <--- 10 200-219 130 130 130 246 246 246 T T F <--- 11 220-239 130 130 130 248 248 248 F F F 12 240-259 121 121 121 224 224 224 T T F <--- 13 260-279 124 124 124 233 233 233 F F F 14 280-299 129 129 129 242 242 242 T T F <--- 15 300-319 130 130 130 246 246 246 F F F 16 320-339 126 126 126 236 236 236 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 130 130 130 246 246 246 F F F 19 380-399 129 129 129 242 242 242 F F F 20 400-419 127 127 127 237 237 237 F F F 21 420-439 127 127 127 237 237 237 T T F <--- 22 440-459 127 127 127 237 237 237 F F F 23 460-479 132 132 131 250 250 249 F F F <--- 24 480-499 131 131 131 249 249 249 F F F 25 500-519 131 131 131 249 249 249 T T F <--- 26 520-539 130 130 130 246 246 246 F F F 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 132, Max flush: 250 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 133 133 133 251 251 251 F F F 1 020-039 133 133 133 251 251 251 F F F 2 040-059 133 133 133 251 251 251 F F F 3 060-079 133 133 133 251 251 251 F F F 4 080-099 124 124 124 231 231 231 F F F 5 100-119 133 133 133 251 251 251 F F F 6 120-139 133 133 133 251 251 251 F F F 7 140-159 128 128 128 238 238 238 F F F 8 160-179 133 133 133 251 251 251 F F F 9 180-199 133 133 133 251 251 251 F F F 10 200-219 133 133 133 251 251 251 F F F 11 220-239 130 130 130 248 248 248 F F F 12 240-259 133 133 133 251 251 251 F F F 13 260-279 124 124 124 233 233 233 F F F 14 280-299 133 133 133 251 251 251 F F F 15 300-319 130 130 130 246 246 246 F F F 16 320-339 126 126 126 236 236 236 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 130 130 130 246 246 246 F F F 19 380-399 129 129 129 242 242 242 F F F 20 400-419 127 127 127 237 237 237 F F F 21 420-439 133 133 133 251 251 251 F F F 22 440-459 127 127 127 237 237 237 F F F 23 460-479 132 132 132 250 250 250 F F F 24 480-499 131 131 131 249 249 249 F F F 25 500-519 133 133 133 251 251 251 F F F 26 520-539 130 130 130 246 246 246 F F F 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 133, Max flush: 251 Begin loop 67 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.149613508Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.157748809Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158065883Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158102588Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158111536Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158123075Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673} {"msg":"Crucible a0b1bdbc-75fd-4303-96a5-12c14b8af026 has session id: 51ce65c2-85f2-4edd-821b-27017df3c408","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158478113Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Upstairs opts: Upstairs UUID: a0b1bdbc-75fd-4303-96a5-12c14b8af026, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158502139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Crucible stats registered with UUID: a0b1bdbc-75fd-4303-96a5-12c14b8af026","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158512018Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:10.158520917Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158670885Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"io task","client":"2","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158693431Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"io task","client":"0","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.15870355Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"io task","client":"1","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158822412Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"task":"crutest"} {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158847869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673} {"msg":"a0b1bdbc-75fd-4303-96a5-12c14b8af026 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158874285Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158883484Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"client":"0","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158893112Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"client":"1","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158902711Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"client":"2","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158912069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"io task","client":"2","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158945404Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"io task","client":"1","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.15897746Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"io task","client":"0","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"ds_connection connected from Ok(127.0.0.1:33262)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.158988948Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"io task","client":"2","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"ds_connection connected from Ok(127.0.0.1:49296)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.159002876Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"io task","client":"1","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"ds_connection connected from Ok(127.0.0.1:35441)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.159075815Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"io task","client":"0","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.159557655Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"client":"0","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.159677328Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"client":"1","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.159784212Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"client":"2","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.15986315Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.159966065Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"[0]R flush_numbers[0..12]: [251, 251, 251, 251, 231, 251, 251, 238, 251, 251, 251, 248]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160051363Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"[0]R generation[0..12]: [133, 133, 133, 133, 124, 133, 133, 128, 133, 133, 133, 130]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160062681Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.16007361Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"[1]R flush_numbers[0..12]: [251, 251, 251, 251, 231, 251, 251, 238, 251, 251, 251, 248]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160131301Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"[1]R generation[0..12]: [133, 133, 133, 133, 124, 133, 133, 128, 133, 133, 133, 130]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.16014197Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160154998Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"[2]R flush_numbers[0..12]: [251, 251, 251, 251, 231, 251, 251, 238, 251, 251, 251, 248]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160165376Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"[2]R generation[0..12]: [133, 133, 133, 133, 124, 133, 133, 128, 133, 133, 133, 130]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160175805Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160186583Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160195952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.16020564Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Next flush: 252","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160214859Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160224128Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"":"downstairs","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160233756Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160242385Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"Wait for a query_work_queue command to finish before sending IO client":"0","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160256443Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"client":"1","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160265961Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"client":"2","":"downstairs","session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.16027564Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160284719Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"a0b1bdbc-75fd-4303-96a5-12c14b8af026 is now active with session: 51ce65c2-85f2-4edd-821b-27017df3c408","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160294677Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160304736Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160312375Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673,"session_id":"51ce65c2-85f2-4edd-821b-27017df3c408"} {"msg":"Activated sub_volume a0b1bdbc-75fd-4303-96a5-12c14b8af026","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:10.160321523Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 294 len 4096 data: 11 02/30 Write block 25 len 28672 data: 9 9 7 10 9 9 7 03/30 Read block 588 len 32768 04/30 Write block 483 len 12288 data: 10 8 8 05/30 Read block 157 len 28672 06/30 Read block 101 len 32768 07/30 Flush 08/30 Write block 14 len 36864 data: 15 13 12 12 11 9 9 8 6 09/30 Write block 402 len 32768 data: 11 10 11 12 12 11 8 7 10/30 Write block 29 len 4096 data: 10 11/30 Write block 234 len 12288 data: 5 4 3 12/30 Write block 261 len 32768 data: 8 11 13 14 14 15 15 13 13/30 Read block 106 len 12288 14/30 Write block 81 len 12288 data: 10 11 10 15/30 Flush 16/30 Flush 17/30 Write block 510 len 16384 data: 11 10 10 9 18/30 Read block 95 len 40960 19/30 Read block 219 len 28672 20/30 Read block 103 len 16384 21/30 Read block 293 len 4096 22/30 Read block 124 len 4096 23/30 Read block 97 len 20480 24/30 Read block 578 len 20480 25/30 Write block 256 len 32768 data: 6 5 5 7 5 9 12 14 26/30 Read block 12 len 40960 27/30 Read block 101 len 28672 28/30 Write block 444 len 12288 data: 12 13 12 29/30 Read block 565 len 28672 30/30 Write block 421 len 36864 data: 8 11 11 10 12 13 13 10 10 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 7 Done Done Sent false 1002 Acked Read 8 Done Done Sent false 1003 Acked Write 3 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Read 8 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Write 3 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Write 3 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Read 4 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Read 5 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Read 10 Done Done Sent false 1026 Acked Read 7 Done Done Sent false 1027 Acked Write 3 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1015)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:10.169141094Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1673} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 133 253 253 251 F F F <--- 1 020-039 134 134 133 253 253 251 F F F <--- 2 040-059 133 133 133 251 251 251 F F F 3 060-079 133 133 133 251 251 251 F F F 4 080-099 134 134 124 253 253 231 F F F <--- 5 100-119 133 133 133 251 251 251 F F F 6 120-139 133 133 133 251 251 251 F F F 7 140-159 128 128 128 238 238 238 F F F 8 160-179 133 133 133 251 251 251 F F F 9 180-199 133 133 133 251 251 251 F F F 10 200-219 133 133 133 251 251 251 F F F 11 220-239 134 134 130 253 253 248 F F F <--- 12 240-259 133 133 133 251 251 251 T T F <--- 13 260-279 134 134 124 253 253 233 T T F <--- 14 280-299 134 134 133 252 252 251 F F T <--- 15 300-319 130 130 130 246 246 246 F F F 16 320-339 126 126 126 236 236 236 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 130 130 130 246 246 246 F F F 19 380-399 129 129 129 242 242 242 F F F 20 400-419 134 134 127 253 253 237 F F F <--- 21 420-439 133 133 133 251 251 251 T T F <--- 22 440-459 127 127 127 237 237 237 T T F <--- 23 460-479 132 132 132 250 250 250 F F F 24 480-499 134 134 131 252 252 249 F F F <--- 25 500-519 133 133 133 251 251 251 T T F <--- 26 520-539 130 130 130 246 246 246 F F F 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 134, Max flush: 253 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 134 134 134 253 253 253 F F F 2 040-059 133 133 133 251 251 251 F F F 3 060-079 133 133 133 251 251 251 F F F 4 080-099 134 134 134 253 253 253 F F F 5 100-119 133 133 133 251 251 251 F F F 6 120-139 133 133 133 251 251 251 F F F 7 140-159 128 128 128 238 238 238 F F F 8 160-179 133 133 133 251 251 251 F F F 9 180-199 133 133 133 251 251 251 F F F 10 200-219 133 133 133 251 251 251 F F F 11 220-239 134 134 134 253 253 253 F F F 12 240-259 135 135 135 254 254 254 F F F 13 260-279 135 135 135 254 254 254 F F F 14 280-299 134 134 134 252 252 252 F F F 15 300-319 130 130 130 246 246 246 F F F 16 320-339 126 126 126 236 236 236 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 130 130 130 246 246 246 F F F 19 380-399 129 129 129 242 242 242 F F F 20 400-419 134 134 134 253 253 253 F F F 21 420-439 135 135 135 254 254 254 F F F 22 440-459 135 135 135 254 254 254 F F F 23 460-479 132 132 132 250 250 250 F F F 24 480-499 134 134 134 252 252 252 F F F 25 500-519 135 135 135 254 254 254 F F F 26 520-539 130 130 130 246 246 246 F F F 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 135, Max flush: 254 Begin loop 68 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.419637103Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.427793831Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428114594Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.42814374Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428152699Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428164087Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681} {"msg":"Crucible a9024263-4763-49c0-8700-6c2add746163 has session id: 841b2df9-05b5-4b5e-898f-457dd4967707","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428543362Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Upstairs opts: Upstairs UUID: a9024263-4763-49c0-8700-6c2add746163, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428599703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Crucible stats registered with UUID: a9024263-4763-49c0-8700-6c2add746163","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428610172Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:14.42861932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428766399Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"io task","client":"1","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428803154Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"io task","client":"0","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428816062Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"io task","client":"2","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428934814Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"task":"crutest"} {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428959841Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681} {"msg":"a9024263-4763-49c0-8700-6c2add746163 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428983837Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.428993186Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"client":"0","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.429011843Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"client":"1","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.429021412Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"client":"2","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.42903101Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"io task","client":"2","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.429041089Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"io task","client":"0","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.429050887Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"io task","client":"1","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"ds_connection connected from Ok(127.0.0.1:37582)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.42909953Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"io task","client":"0","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"ds_connection connected from Ok(127.0.0.1:61151)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.429125526Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"io task","client":"1","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"ds_connection connected from Ok(127.0.0.1:63694)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.429137035Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"io task","client":"2","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.429817795Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"client":"0","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.429862269Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"client":"2","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.42998743Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"client":"1","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430114882Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430177683Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"[0]R flush_numbers[0..12]: [253, 253, 251, 251, 253, 251, 251, 238, 251, 251, 251, 253]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430240703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"[0]R generation[0..12]: [134, 134, 133, 133, 134, 133, 133, 128, 133, 133, 133, 134]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430252302Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.43026325Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"[1]R flush_numbers[0..12]: [253, 253, 251, 251, 253, 251, 251, 238, 251, 251, 251, 253]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430273989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"[1]R generation[0..12]: [134, 134, 133, 133, 134, 133, 133, 128, 133, 133, 133, 134]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430284297Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430295325Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"[2]R flush_numbers[0..12]: [253, 253, 251, 251, 253, 251, 251, 238, 251, 251, 251, 253]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430305744Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"[2]R generation[0..12]: [134, 134, 133, 133, 134, 133, 133, 128, 133, 133, 133, 134]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430320882Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.43033174Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430341299Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430350977Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Next flush: 255","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430360366Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430372234Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"":"downstairs","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430382033Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430390741Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"client":"0","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.43040058Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"client":"1","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430410419Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"client":"2","":"downstairs","session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430420117Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430429206Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"Wait for a query_work_queue command to finish before sending IO session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"a9024263-4763-49c0-8700-6c2add746163 is now active with session: 841b2df9-05b5-4b5e-898f-457dd4967707","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430443704Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430452642Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.430460051Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681,"session_id":"841b2df9-05b5-4b5e-898f-457dd4967707"} {"msg":"Activated sub_volume a9024263-4763-49c0-8700-6c2add746163","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:14.43046909Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 513 len 28672 data: 10 12 10 8 10 8 9 02/30 Write block 96 len 16384 data: 12 15 14 15 03/30 Read block 16 len 28672 04/30 Write block 124 len 16384 data: 9 8 10 10 05/30 Read block 590 len 12288 06/30 Read block 110 len 40960 07/30 Read block 450 len 16384 08/30 Read block 296 len 32768 09/30 Write block 30 len 4096 data: 10 10/30 Read block 500 len 40960 11/30 Write block 386 len 32768 data: 7 8 10 10 9 13 13 13 12/30 Read block 425 len 36864 13/30 Read block 225 len 16384 14/30 Write block 431 len 16384 data: 13 15 17 19 15/30 Read block 355 len 16384 16/30 Read block 499 len 32768 17/30 Write block 77 len 28672 data: 6 8 6 6 11 12 11 18/30 Read block 195 len 28672 19/30 Read block 492 len 4096 20/30 Read block 588 len 4096 21/30 Write block 433 len 28672 data: 18 20 20 19 17 19 14 22/30 Write block 302 len 24576 data: 6 7 9 9 12 8 23/30 Read block 240 len 8192 24/30 Read block 55 len 4096 25/30 Read block 142 len 4096 26/30 Read block 227 len 8192 27/30 Read block 155 len 32768 28/30 Read block 330 len 20480 29/30 Write block 319 len 8192 data: 14 13 30/30 Read block 181 len 12288 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Write 1 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 4 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Read 2 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 2 Done Sent Done false 1029 Acked Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 0 27 Done 30 3 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:14.43950247Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1681} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 134 134 134 253 253 253 T F T <--- 2 040-059 133 133 133 251 251 251 F F F 3 060-079 133 133 133 251 251 251 T F T <--- 4 080-099 134 134 134 253 253 253 T T T 5 100-119 133 133 133 251 251 251 F F F 6 120-139 133 133 133 251 251 251 T F T <--- 7 140-159 128 128 128 238 238 238 F F F 8 160-179 133 133 133 251 251 251 F F F 9 180-199 133 133 133 251 251 251 F F F 10 200-219 133 133 133 251 251 251 F F F 11 220-239 134 134 134 253 253 253 F F F 12 240-259 135 135 135 254 254 254 F F F 13 260-279 135 135 135 254 254 254 F F F 14 280-299 134 134 134 252 252 252 F F F 15 300-319 130 130 130 246 246 246 T F T <--- 16 320-339 126 126 126 236 236 236 T F T <--- 17 340-359 118 118 118 220 220 220 F F F 18 360-379 130 130 130 246 246 246 F F F 19 380-399 129 129 129 242 242 242 T F T <--- 20 400-419 134 134 134 253 253 253 F F F 21 420-439 135 135 135 254 254 254 T F T <--- 22 440-459 135 135 135 254 254 254 F F F 23 460-479 132 132 132 250 250 250 F F F 24 480-499 134 134 134 252 252 252 F F F 25 500-519 135 135 135 254 254 254 T T T 26 520-539 130 130 130 246 246 246 F F F 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 135, Max flush: 254 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 136 136 136 255 255 255 F F F 2 040-059 133 133 133 251 251 251 F F F 3 060-079 136 136 136 255 255 255 F F F 4 080-099 136 136 136 255 255 255 F F F 5 100-119 133 133 133 251 251 251 F F F 6 120-139 136 136 136 255 255 255 F F F 7 140-159 128 128 128 238 238 238 F F F 8 160-179 133 133 133 251 251 251 F F F 9 180-199 133 133 133 251 251 251 F F F 10 200-219 133 133 133 251 251 251 F F F 11 220-239 134 134 134 253 253 253 F F F 12 240-259 135 135 135 254 254 254 F F F 13 260-279 135 135 135 254 254 254 F F F 14 280-299 134 134 134 252 252 252 F F F 15 300-319 136 136 136 255 255 255 F F F 16 320-339 136 136 136 255 255 255 F F F 17 340-359 118 118 118 220 220 220 F F F 18 360-379 130 130 130 246 246 246 F F F 19 380-399 136 136 136 255 255 255 F F F 20 400-419 134 134 134 253 253 253 F F F 21 420-439 136 136 136 255 255 255 F F F 22 440-459 135 135 135 254 254 254 F F F 23 460-479 132 132 132 250 250 250 F F F 24 480-499 134 134 134 252 252 252 F F F 25 500-519 136 136 136 255 255 255 F F F 26 520-539 130 130 130 246 246 246 F F F 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 136, Max flush: 255 Begin loop 69 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.686576939Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:18.693724914Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.693788765Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.700781743Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701047404Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.7010744Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701087468Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701099006Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689} {"msg":"Crucible ac7ca6ba-e30f-4c63-8c04-9e6546d05444 has session id: 35eae206-1391-4430-8cd3-a598ef9f58e4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701434267Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Upstairs opts: Upstairs UUID: ac7ca6ba-e30f-4c63-8c04-9e6546d05444, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701456824Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Crucible stats registered with UUID: ac7ca6ba-e30f-4c63-8c04-9e6546d05444","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701466643Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:18.701475701Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701631059Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"io task","client":"1","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701673632Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"io task","client":"0","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701685031Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"io task","client":"2","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701774738Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"task":"crutest"} {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701799914Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689} {"msg":"ac7ca6ba-e30f-4c63-8c04-9e6546d05444 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.70182746Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701836978Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"client":"0","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701846667Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"client":"1","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701858155Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"client":"2","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701867644Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"io task","client":"2","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701877563Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"io task","client":"0","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701887301Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"io task","client":"1","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:41991)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701949192Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"io task","client":"1","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:52926)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.701975338Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"io task","client":"0","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:61073)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.70210004Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"io task","client":"2","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.702639411Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"client":"2","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.702659018Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"client":"1","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.702698303Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"client":"0","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703007767Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.70305651Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"[0]R flush_numbers[0..12]: [253, 255, 251, 255, 255, 251, 255, 238, 251, 251, 251, 253]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703106643Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"[0]R generation[0..12]: [134, 136, 133, 136, 136, 133, 136, 128, 133, 133, 133, 134]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703117881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.70312874Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"[1]R flush_numbers[0..12]: [253, 255, 251, 255, 255, 251, 255, 238, 251, 251, 251, 253]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703141708Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"[1]R generation[0..12]: [134, 136, 133, 136, 136, 133, 136, 128, 133, 133, 133, 134]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703152096Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703162795Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"[2]R flush_numbers[0..12]: [253, 255, 251, 255, 255, 251, 255, 238, 251, 251, 251, 253]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703173163Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"[2]R generation[0..12]: [134, 136, 133, 136, 136, 133, 136, 128, 133, 133, 133, 134]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703183442Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703194Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Max found gen is 137","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703203399Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Generation requested: 138 >= found:137","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703212977Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Next flush: 256","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703222106Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703231285Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"":"downstairs","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703240693Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703249162Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"client":"0","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703258511Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"client":"1","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703268069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"client":"2","":"downstairs","session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703277728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703288116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"ac7ca6ba-e30f-4c63-8c04-9e6546d05444 is now active with session: 35eae206-1391-4430-8cd3-a598ef9f58e4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703297935Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703306914Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703314303Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689,"session_id":"35eae206-1391-4430-8cd3-a598ef9f58e4"} {"msg":"Activated sub_volume ac7ca6ba-e30f-4c63-8c04-9e6546d05444","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:18.703323401Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 124 len 32768 data: 10 9 11 11 9 10 11 11 02/30 Read block 64 len 28672 03/30 Write block 372 len 36864 data: 8 10 11 10 13 14 11 13 11 04/30 Write block 301 len 4096 data: 5 05/30 Read block 257 len 12288 06/30 Flush 07/30 Write block 124 len 20480 data: 11 10 12 12 10 08/30 Write block 236 len 36864 data: 4 6 10 10 10 13 14 14 14 09/30 Read block 547 len 32768 10/30 Flush 11/30 Write block 180 len 8192 data: 17 13 12/30 Write block 445 len 16384 data: 14 13 11 11 13/30 Write block 233 len 36864 data: 8 6 5 5 7 11 11 11 14 14/30 Read block 171 len 28672 15/30 Write block 72 len 4096 data: 12 16/30 Write block 186 len 4096 data: 9 17/30 Read block 213 len 8192 18/30 Read block 498 len 16384 19/30 Read block 114 len 40960 20/30 Read block 377 len 16384 21/30 Read block 359 len 16384 22/30 Write block 215 len 8192 data: 5 6 23/30 Read block 431 len 28672 24/30 Write block 347 len 36864 data: 12 8 9 10 10 12 12 12 12 25/30 Read block 411 len 28672 26/30 Write block 23 len 4096 data: 6 27/30 Write block 54 len 24576 data: 10 10 13 12 11 10 28/30 Read block 463 len 24576 29/30 Read block 330 len 32768 30/30 Read block 156 len 36864 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Write 1 Sent Done Done false 1004 Acked Read 3 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Read 4 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Read 7 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 1 29 Done 2 30 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1009)) Some(JobId(1009)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:18.712747634Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1689} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 136 136 136 255 255 255 F T T <--- 2 040-059 133 133 133 251 251 251 F T T <--- 3 060-079 136 136 136 255 255 255 F T T <--- 4 080-099 136 136 136 255 255 255 F F F 5 100-119 133 133 133 251 251 251 F F F 6 120-139 136 138 138 255 257 257 T F F <--- 7 140-159 128 128 128 238 238 238 F F F 8 160-179 133 133 133 251 251 251 F F F 9 180-199 133 133 133 251 251 251 F T T <--- 10 200-219 133 133 133 251 251 251 F T T <--- 11 220-239 134 138 138 253 257 257 F T T <--- 12 240-259 135 138 138 254 257 257 F T T <--- 13 260-279 135 135 135 254 254 254 F F F 14 280-299 134 134 134 252 252 252 F F F 15 300-319 136 138 138 255 256 256 F F F <--- 16 320-339 136 136 136 255 255 255 F F F 17 340-359 118 118 118 220 220 220 F T T <--- 18 360-379 130 138 138 246 256 256 F F F <--- 19 380-399 136 138 138 255 256 256 F F F <--- 20 400-419 134 134 134 253 253 253 F F F 21 420-439 136 136 136 255 255 255 F F F 22 440-459 135 135 135 254 254 254 F T T <--- 23 460-479 132 132 132 250 250 250 F F F 24 480-499 134 134 134 252 252 252 F F F 25 500-519 136 136 136 255 255 255 F F F 26 520-539 130 130 130 246 246 246 F F F 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 138, Max flush: 257 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 139 139 139 258 258 258 F F F 2 040-059 139 139 139 258 258 258 F F F 3 060-079 139 139 139 258 258 258 F F F 4 080-099 136 136 136 255 255 255 F F F 5 100-119 133 133 133 251 251 251 F F F 6 120-139 138 138 138 257 257 257 F F F 7 140-159 128 128 128 238 238 238 F F F 8 160-179 133 133 133 251 251 251 F F F 9 180-199 139 139 139 258 258 258 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 139 139 139 258 258 258 F F F 12 240-259 139 139 139 258 258 258 F F F 13 260-279 135 135 135 254 254 254 F F F 14 280-299 134 134 134 252 252 252 F F F 15 300-319 138 138 138 256 256 256 F F F 16 320-339 136 136 136 255 255 255 F F F 17 340-359 139 139 139 258 258 258 F F F 18 360-379 138 138 138 256 256 256 F F F 19 380-399 138 138 138 256 256 256 F F F 20 400-419 134 134 134 253 253 253 F F F 21 420-439 136 136 136 255 255 255 F F F 22 440-459 139 139 139 258 258 258 F F F 23 460-479 132 132 132 250 250 250 F F F 24 480-499 134 134 134 252 252 252 F F F 25 500-519 136 136 136 255 255 255 F F F 26 520-539 130 130 130 246 246 246 F F F 27 540-559 128 128 128 241 241 241 F F F 28 560-579 118 118 118 220 220 220 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 139, Max flush: 258 Begin loop 70 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.97578212Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:22.982995995Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.983060166Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.989897647Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990168827Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990203932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.9902132Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990227238Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697} {"msg":"Crucible b7ce7538-5ef2-4113-be39-f76e9d596bf5 has session id: 8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990606173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Upstairs opts: Upstairs UUID: b7ce7538-5ef2-4113-be39-f76e9d596bf5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990635029Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Crucible stats registered with UUID: b7ce7538-5ef2-4113-be39-f76e9d596bf5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990647997Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:22.990657196Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990799605Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"io task","client":"2","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.99083522Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"io task","client":"0","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990846388Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"io task","client":"1","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990932595Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"task":"crutest"} {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990949693Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697} {"msg":"b7ce7538-5ef2-4113-be39-f76e9d596bf5 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990983058Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.990992437Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"client":"0","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.991002345Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"client":"1","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.991012134Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"client":"2","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.991021662Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"io task","client":"2","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.991031671Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"io task","client":"1","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"ds_connection connected from Ok(127.0.0.1:53768)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.991057627Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"io task","client":"2","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.991068146Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"io task","client":"0","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"ds_connection connected from Ok(127.0.0.1:59535)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.991131006Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"io task","client":"1","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"ds_connection connected from Ok(127.0.0.1:62659)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.991144344Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"io task","client":"0","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.99178732Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"client":"1","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.991810827Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"client":"2","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.991825125Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"client":"0","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992113103Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992156956Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"[0]R flush_numbers[0..12]: [253, 258, 258, 258, 255, 251, 257, 238, 251, 258, 258, 258]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992210579Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"[0]R generation[0..12]: [134, 139, 139, 139, 136, 133, 138, 128, 133, 139, 139, 139]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992221657Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992232645Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"[1]R flush_numbers[0..12]: [253, 258, 258, 258, 255, 251, 257, 238, 251, 258, 258, 258]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992243214Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"[1]R generation[0..12]: [134, 139, 139, 139, 136, 133, 138, 128, 133, 139, 139, 139]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992253612Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992264381Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"[2]R flush_numbers[0..12]: [253, 258, 258, 258, 255, 251, 257, 238, 251, 258, 258, 258]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992274949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"[2]R generation[0..12]: [134, 139, 139, 139, 136, 133, 138, 128, 133, 139, 139, 139]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992287587Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992298626Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Max found gen is 140","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992308034Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992317683Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Next flush: 259","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992326962Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.99233625Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"":"downstairs","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992345979Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992354747Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"client":"0","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992364316Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"client":"1","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992373725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"client":"2","":"downstairs","session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992383563Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:15:22.992392432Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"b7ce7538-5ef2-4113-be39-f76e9d596bf5 is now active with session: 8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.99240597Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992415079Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992422688Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697,"session_id":"8dff8181-cd9a-40e0-adbf-c0fb58d0d0dd"} {"msg":"Activated sub_volume b7ce7538-5ef2-4113-be39-f76e9d596bf5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:22.992431736Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 163 len 28672 data: 18 20 19 16 15 12 11 02/30 Read block 412 len 8192 03/30 Read block 501 len 28672 04/30 Write block 557 len 32768 data: 14 13 11 12 12 10 10 11 05/30 Write block 535 len 4096 data: 15 06/30 Flush 07/30 Write block 415 len 36864 data: 6 6 7 8 8 8 9 12 12 08/30 Write block 98 len 4096 data: 15 09/30 Flush 10/30 Write block 427 len 4096 data: 14 11/30 Read block 494 len 4096 12/30 Read block 448 len 28672 13/30 Read block 504 len 8192 14/30 Read block 156 len 24576 15/30 Write block 349 len 32768 data: 10 11 11 13 13 13 13 13 16/30 Flush 17/30 Flush 18/30 Write block 413 len 24576 data: 8 7 7 7 8 9 19/30 Write block 319 len 32768 data: 15 14 8 7 7 7 8 8 20/30 Write block 180 len 20480 data: 18 14 14 14 11 21/30 Flush 22/30 Write block 43 len 8192 data: 15 12 23/30 Read block 248 len 20480 24/30 Read block 420 len 4096 25/30 Write block 132 len 36864 data: 12 11 9 7 7 8 9 14 18 26/30 Flush 27/30 Write block 488 len 4096 data: 7 28/30 Write block 52 len 24576 data: 10 9 11 11 14 13 29/30 Read block 188 len 4096 30/30 Write block 355 len 16384 data: 14 14 13 11 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Sent Done Done false 1001 Acked Read 2 Sent Done Done false 1002 Acked Read 7 Sent Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Write 1 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 9 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 1 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 5 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:23.001616404Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1697} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 139 139 139 258 258 258 F F F 2 040-059 139 140 140 258 264 264 F T T <--- 3 060-079 139 139 139 258 258 258 F F F 4 080-099 136 140 140 255 260 260 F F F <--- 5 100-119 133 133 133 251 251 251 F F F 6 120-139 138 140 140 257 264 264 F F F <--- 7 140-159 128 140 140 238 264 264 F F F <--- 8 160-179 133 140 140 251 259 259 F F F <--- 9 180-199 139 140 140 258 263 263 F F F <--- 10 200-219 139 139 139 258 258 258 F F F 11 220-239 139 139 139 258 258 258 F F F 12 240-259 139 139 139 258 258 258 F F F 13 260-279 135 135 135 254 254 254 F F F 14 280-299 134 134 134 252 252 252 F F F 15 300-319 138 140 140 256 263 263 F F F <--- 16 320-339 136 140 140 255 263 263 F F F <--- 17 340-359 139 140 140 258 261 261 F T T <--- 18 360-379 138 138 138 256 256 256 F F F 19 380-399 138 138 138 256 256 256 F F F 20 400-419 134 140 140 253 263 263 F F F <--- 21 420-439 136 140 140 255 261 261 F F F <--- 22 440-459 139 139 139 258 258 258 F F F 23 460-479 132 132 132 250 250 250 F F F 24 480-499 134 134 134 252 252 252 F T T <--- 25 500-519 136 136 136 255 255 255 F F F 26 520-539 130 140 140 246 259 259 F F F <--- 27 540-559 128 140 140 241 259 259 F F F <--- 28 560-579 118 140 140 220 259 259 F F F <--- 29 580-599 123 123 123 229 229 229 F F F Max gen: 140, Max flush: 264 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 139 139 139 258 258 258 F F F 2 040-059 141 141 141 265 265 265 F F F 3 060-079 139 139 139 258 258 258 F F F 4 080-099 140 140 140 260 260 260 F F F 5 100-119 133 133 133 251 251 251 F F F 6 120-139 140 140 140 264 264 264 F F F 7 140-159 140 140 140 264 264 264 F F F 8 160-179 140 140 140 259 259 259 F F F 9 180-199 140 140 140 263 263 263 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 139 139 139 258 258 258 F F F 12 240-259 139 139 139 258 258 258 F F F 13 260-279 135 135 135 254 254 254 F F F 14 280-299 134 134 134 252 252 252 F F F 15 300-319 140 140 140 263 263 263 F F F 16 320-339 140 140 140 263 263 263 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 138 138 138 256 256 256 F F F 19 380-399 138 138 138 256 256 256 F F F 20 400-419 140 140 140 263 263 263 F F F 21 420-439 140 140 140 261 261 261 F F F 22 440-459 139 139 139 258 258 258 F F F 23 460-479 132 132 132 250 250 250 F F F 24 480-499 141 141 141 265 265 265 F F F 25 500-519 136 136 136 255 255 255 F F F 26 520-539 140 140 140 259 259 259 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 140 140 140 259 259 259 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 141, Max flush: 265 Begin loop 71 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.266451897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.274831162Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275113571Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275139237Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275147896Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275159234Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705} {"msg":"Crucible cfc71d27-9ff5-48af-8c13-43fbab1b3c62 has session id: 95eec8ff-0ab3-474d-9af1-77a3698b98a4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275493495Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Upstairs opts: Upstairs UUID: cfc71d27-9ff5-48af-8c13-43fbab1b3c62, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275516462Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Crucible stats registered with UUID: cfc71d27-9ff5-48af-8c13-43fbab1b3c62","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275526481Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:27.275535629Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275713493Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"io task","client":"0","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275754057Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"io task","client":"1","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275766166Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"io task","client":"2","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275847594Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"task":"crutest"} {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275869131Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705} {"msg":"cfc71d27-9ff5-48af-8c13-43fbab1b3c62 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275890967Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275900556Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"client":"0","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275911104Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"client":"1","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275921973Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"client":"2","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275932611Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"io task","client":"2","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.275949039Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"io task","client":"1","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:61480)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.276014519Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"io task","client":"2","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.276136411Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"io task","client":"0","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:34573)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.276150769Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"io task","client":"1","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:44936)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.276161218Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"io task","client":"0","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.276723886Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"client":"0","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.276736854Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"client":"1","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.276747612Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"client":"2","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277028171Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.27703769Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"[0]R flush_numbers[0..12]: [253, 258, 265, 258, 260, 251, 264, 264, 259, 263, 258, 258]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277055627Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"[0]R generation[0..12]: [134, 139, 141, 139, 140, 133, 140, 140, 140, 140, 139, 139]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277066326Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277077184Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"[1]R flush_numbers[0..12]: [253, 258, 265, 258, 260, 251, 264, 264, 259, 263, 258, 258]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277087652Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"[1]R generation[0..12]: [134, 139, 141, 139, 140, 133, 140, 140, 140, 140, 139, 139]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277098021Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277108929Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"[2]R flush_numbers[0..12]: [253, 258, 265, 258, 260, 251, 264, 264, 259, 263, 258, 258]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277119518Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"[2]R generation[0..12]: [134, 139, 141, 139, 140, 133, 140, 140, 140, 140, 139, 139]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277130026Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277140695Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Max found gen is 142","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277149973Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277159532Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Next flush: 266","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277168631Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277177729Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"":"downstairs","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277187528Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277196227Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"client":"0","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277207005Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"client":"1","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277216414Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"client":"2","":"downstairs","session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277226112Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277235111Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Wait for a query_work_queue command to finish before sending IO cfc71d27-9ff5-48af-8c13-43fbab1b3c62 is now active with session: 95eec8ff-0ab3-474d-9af1-77a3698b98a4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277248829Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277257958Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277265496Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705,"session_id":"95eec8ff-0ab3-474d-9af1-77a3698b98a4"} {"msg":"Activated sub_volume cfc71d27-9ff5-48af-8c13-43fbab1b3c62","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:27.277274565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 138 len 36864 data: 10 15 19 17 15 16 14 14 11 02/30 Write block 181 len 12288 data: 15 15 15 03/30 Flush 04/30 Flush 05/30 Write block 134 len 8192 data: 10 8 06/30 Write block 441 len 36864 data: 13 14 12 13 15 14 12 12 13 07/30 Read block 572 len 40960 08/30 Read block 486 len 32768 09/30 Read block 53 len 8192 10/30 Read block 467 len 4096 11/30 Read block 481 len 32768 12/30 Write block 76 len 40960 data: 10 7 9 7 7 12 13 12 12 14 13/30 Read block 174 len 32768 14/30 Flush 15/30 Write block 149 len 20480 data: 7 6 7 6 5 16/30 Read block 69 len 40960 17/30 Write block 570 len 12288 data: 12 13 14 18/30 Read block 393 len 16384 19/30 Read block 265 len 8192 20/30 Read block 324 len 32768 21/30 Write block 273 len 4096 data: 12 22/30 Write block 288 len 16384 data: 11 10 11 11 23/30 Read block 388 len 24576 24/30 Flush 25/30 Flush 26/30 Write block 514 len 36864 data: 13 11 9 11 9 10 9 11 11 27/30 Write block 473 len 28672 data: 17 16 13 8 6 5 5 28/30 Read block 533 len 4096 29/30 Read block 522 len 36864 30/30 Read block 315 len 12288 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Sent Done false 1001 Acked Write 3 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 2 Done Sent Done false 1005 Acked Write 9 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Read 8 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:27.286428757Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1705} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 139 139 139 258 258 258 F F F 2 040-059 141 141 141 265 265 265 F F F 3 060-079 142 139 142 268 258 268 F F F <--- 4 080-099 142 140 142 268 260 268 F F F <--- 5 100-119 133 133 133 251 251 251 F F F 6 120-139 142 140 142 268 264 268 F F F <--- 7 140-159 142 140 142 269 264 269 F F F <--- 8 160-179 140 140 140 259 259 259 F F F 9 180-199 142 140 142 266 263 266 F F F <--- 10 200-219 139 139 139 258 258 258 F F F 11 220-239 139 139 139 258 258 258 F F F 12 240-259 139 139 139 258 258 258 F F F 13 260-279 142 135 142 269 254 269 F F F <--- 14 280-299 142 134 142 269 252 269 F F F <--- 15 300-319 140 140 140 263 263 263 F F F 16 320-339 140 140 140 263 263 263 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 138 138 138 256 256 256 F F F 19 380-399 138 138 138 256 256 256 F F F 20 400-419 140 140 140 263 263 263 F F F 21 420-439 140 140 140 261 261 261 F F F 22 440-459 142 139 142 268 258 268 F F F <--- 23 460-479 132 132 132 250 250 250 T F T <--- 24 480-499 141 141 141 265 265 265 F F F 25 500-519 136 136 136 255 255 255 T F T <--- 26 520-539 140 140 140 259 259 259 T F T <--- 27 540-559 140 140 140 259 259 259 F F F 28 560-579 142 140 142 269 259 269 F F F <--- 29 580-599 123 123 123 229 229 229 F F F Max gen: 142, Max flush: 269 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 139 139 139 258 258 258 F F F 2 040-059 141 141 141 265 265 265 F F F 3 060-079 142 142 142 268 268 268 F F F 4 080-099 142 142 142 268 268 268 F F F 5 100-119 133 133 133 251 251 251 F F F 6 120-139 142 142 142 268 268 268 F F F 7 140-159 142 142 142 269 269 269 F F F 8 160-179 140 140 140 259 259 259 F F F 9 180-199 142 142 142 266 266 266 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 139 139 139 258 258 258 F F F 12 240-259 139 139 139 258 258 258 F F F 13 260-279 142 142 142 269 269 269 F F F 14 280-299 142 142 142 269 269 269 F F F 15 300-319 140 140 140 263 263 263 F F F 16 320-339 140 140 140 263 263 263 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 138 138 138 256 256 256 F F F 19 380-399 138 138 138 256 256 256 F F F 20 400-419 140 140 140 263 263 263 F F F 21 420-439 140 140 140 261 261 261 F F F 22 440-459 142 142 142 268 268 268 F F F 23 460-479 143 143 143 270 270 270 F F F 24 480-499 141 141 141 265 265 265 F F F 25 500-519 143 143 143 270 270 270 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 142 142 142 269 269 269 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 143, Max flush: 270 Begin loop 72 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.537001665Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:31.544261624Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.544314956Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.551323422Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.551584554Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.551612909Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.551621428Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.551632847Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713} {"msg":"Crucible d7bef3e0-2c13-4de6-8ef0-4e85dc0bc8db has session id: 5eb9faf6-ee23-4b8a-88a8-365ebb96e667","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.551967098Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Upstairs opts: Upstairs UUID: d7bef3e0-2c13-4de6-8ef0-4e85dc0bc8db, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.551995603Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Crucible stats registered with UUID: d7bef3e0-2c13-4de6-8ef0-4e85dc0bc8db","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.55201749Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:31.552026979Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.55215439Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"io task","client":"2","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552178467Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"io task","client":"0","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552188495Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"io task","client":"1","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552342973Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"task":"crutest"} {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552367269Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713} {"msg":"d7bef3e0-2c13-4de6-8ef0-4e85dc0bc8db active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552393245Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552402484Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"client":"0","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552412103Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"client":"1","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552421711Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"client":"2","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.55243108Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"io task","client":"2","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552440958Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"io task","client":"0","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552452167Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"io task","client":"1","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"ds_connection connected from Ok(127.0.0.1:47982)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552487292Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"io task","client":"2","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"ds_connection connected from Ok(127.0.0.1:63770)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552502769Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"io task","client":"1","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"ds_connection connected from Ok(127.0.0.1:57612)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.552513208Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"io task","client":"0","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553149875Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"client":"2","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553172981Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"client":"1","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553226944Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"client":"0","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553468878Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553478517Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"[0]R flush_numbers[0..12]: [253, 258, 265, 268, 268, 251, 268, 269, 259, 266, 258, 258]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553538848Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"[0]R generation[0..12]: [134, 139, 141, 142, 142, 133, 142, 142, 140, 142, 139, 139]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553550116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553561005Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"[1]R flush_numbers[0..12]: [253, 258, 265, 268, 268, 251, 268, 269, 259, 266, 258, 258]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553571513Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"[1]R generation[0..12]: [134, 139, 141, 142, 142, 133, 142, 142, 140, 142, 139, 139]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553581832Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.55359518Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"[2]R flush_numbers[0..12]: [253, 258, 265, 268, 268, 251, 268, 269, 259, 266, 258, 258]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553605798Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"[2]R generation[0..12]: [134, 139, 141, 142, 142, 133, 142, 142, 140, 142, 139, 139]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553616097Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553626865Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Max found gen is 144","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553636264Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Generation requested: 144 >= found:144","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553645982Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Next flush: 271","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553655251Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.55366449Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"":"downstairs","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553673998Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553682537Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"client":"0","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553692305Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"client":"1","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553702204Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"client":"2","":"downstairs","session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553712233Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} Wait for a query_work_queue command to finish before sending IO {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553725451Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"d7bef3e0-2c13-4de6-8ef0-4e85dc0bc8db is now active with session: 5eb9faf6-ee23-4b8a-88a8-365ebb96e667","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553735449Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553745088Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713,"session_id":"5eb9faf6-ee23-4b8a-88a8-365ebb96e667"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553754216Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713} {"msg":"Activated sub_volume d7bef3e0-2c13-4de6-8ef0-4e85dc0bc8db","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:31.553762115Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 317 len 12288 data: 16 15 16 02/30 Read block 434 len 12288 03/30 Read block 361 len 40960 04/30 Write block 388 len 12288 data: 11 11 10 05/30 Flush 06/30 Write block 38 len 32768 data: 12 13 14 15 15 16 13 12 07/30 Write block 33 len 24576 data: 6 5 7 7 11 13 08/30 Write block 245 len 12288 data: 13 14 12 09/30 Write block 148 len 24576 data: 8 8 7 8 7 6 10/30 Read block 277 len 24576 11/30 Read block 66 len 20480 12/30 Read block 204 len 4096 13/30 Write block 151 len 8192 data: 9 8 14/30 Read block 333 len 40960 15/30 Write block 499 len 4096 data: 11 16/30 Write block 62 len 28672 data: 11 9 12 14 11 11 10 17/30 Write block 95 len 16384 data: 11 13 16 16 18/30 Read block 42 len 20480 19/30 Write block 242 len 36864 data: 15 15 15 14 15 13 11 10 11 20/30 Write block 264 len 32768 data: 15 15 16 16 14 14 11 12 21/30 Read block 463 len 20480 22/30 Read block 490 len 28672 23/30 Flush 24/30 Write block 27 len 4096 data: 8 25/30 Read block 323 len 40960 26/30 Read block 483 len 24576 27/30 Read block 176 len 4096 28/30 Write block 279 len 36864 data: 12 11 12 9 11 11 12 11 9 29/30 Read block 431 len 4096 30/30 Write block 221 len 24576 data: 6 9 13 11 12 14 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 8 Sent Done Done false 1006 Acked Write 6 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Write 6 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Read 5 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 9 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 1 1 28 Done 4 29 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:31.562925346Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1713} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 139 144 144 258 272 272 F T T <--- 2 040-059 141 144 144 265 272 272 F F F <--- 3 060-079 142 144 144 268 272 272 F F F <--- 4 080-099 142 144 144 268 272 272 F F F <--- 5 100-119 133 133 133 251 251 251 F F F 6 120-139 142 142 142 268 268 268 F F F 7 140-159 142 144 144 269 272 272 F F F <--- 8 160-179 140 140 140 259 259 259 F F F 9 180-199 142 142 142 266 266 266 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 139 139 139 258 258 258 F T T <--- 12 240-259 139 144 144 258 272 272 F F F <--- 13 260-279 142 144 144 269 272 272 F T T <--- 14 280-299 142 142 142 269 269 269 F T T <--- 15 300-319 140 144 144 263 271 271 T F F <--- 16 320-339 140 140 140 263 263 263 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 138 138 138 256 256 256 F F F 19 380-399 138 144 144 256 271 271 T F F <--- 20 400-419 140 140 140 263 263 263 F F F 21 420-439 140 140 140 261 261 261 F F F 22 440-459 142 142 142 268 268 268 F F F 23 460-479 143 143 143 270 270 270 F F F 24 480-499 141 144 144 265 272 272 F F F <--- 25 500-519 143 143 143 270 270 270 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 142 142 142 269 269 269 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 144, Max flush: 272 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 72 Verifying data now No change after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 145 145 145 273 273 273 F F F 2 040-059 144 144 144 272 272 272 F F F 3 060-079 144 144 144 272 272 272 F F F 4 080-099 144 144 144 272 272 272 F F F 5 100-119 133 133 133 251 251 251 F F F 6 120-139 142 142 142 268 268 268 F F F 7 140-159 144 144 144 272 272 272 F F F 8 160-179 140 140 140 259 259 259 F F F 9 180-199 142 142 142 266 266 266 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 145 145 145 273 273 273 F F F 12 240-259 144 144 144 272 272 272 F F F 13 260-279 145 145 145 273 273 273 F F F 14 280-299 145 145 145 273 273 273 F F F 15 300-319 144 144 144 271 271 271 F F F 16 320-339 140 140 140 263 263 263 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 138 138 138 256 256 256 F F F 19 380-399 144 144 144 271 271 271 F F F 20 400-419 140 140 140 263 263 263 F F F 21 420-439 140 140 140 261 261 261 F F F 22 440-459 142 142 142 268 268 268 F F F 23 460-479 143 143 143 270 270 270 F F F 24 480-499 144 144 144 272 272 272 F F F 25 500-519 143 143 143 270 270 270 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 142 142 142 269 269 269 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 145, Max flush: 273 Begin loop 73 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.818881257Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:35.826055988Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.826119478Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.832938132Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833210382Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833238148Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833246617Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833257855Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721} {"msg":"Crucible 33769ff1-23d7-450d-9c08-29c3d1cd4fd3 has session id: e9e1d7e1-70af-4772-8161-903a982b4c27","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833612483Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Upstairs opts: Upstairs UUID: 33769ff1-23d7-450d-9c08-29c3d1cd4fd3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.83363583Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Crucible stats registered with UUID: 33769ff1-23d7-450d-9c08-29c3d1cd4fd3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833645998Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:35.833654937Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833822293Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"io task","client":"1","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833844679Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"io task","client":"0","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833854468Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"io task","client":"2","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833979949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"task":"crutest"} {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.833997147Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721} {"msg":"33769ff1-23d7-450d-9c08-29c3d1cd4fd3 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834013955Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834022723Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"client":"0","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834032342Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"client":"1","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.8340418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"client":"2","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834053069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"io task","client":"2","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834062977Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"io task","client":"0","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834072766Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"io task","client":"1","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"ds_connection connected from Ok(127.0.0.1:46277)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834132287Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"io task","client":"0","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"ds_connection connected from Ok(127.0.0.1:56940)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834147115Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"io task","client":"1","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"ds_connection connected from Ok(127.0.0.1:34886)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834157334Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"io task","client":"2","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834882927Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"client":"0","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834924741Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"client":"2","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.834945648Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"client":"1","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835245604Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835279889Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"[0]R flush_numbers[0..12]: [253, 273, 272, 272, 272, 251, 268, 272, 259, 266, 258, 273]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835326063Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"[0]R generation[0..12]: [134, 145, 144, 144, 144, 133, 142, 144, 140, 142, 139, 145]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835337281Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835348229Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"[1]R flush_numbers[0..12]: [253, 273, 272, 272, 272, 251, 268, 272, 259, 266, 258, 273]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835358868Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"[1]R generation[0..12]: [134, 145, 144, 144, 144, 133, 142, 144, 140, 142, 139, 145]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835369426Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835380275Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"[2]R flush_numbers[0..12]: [253, 273, 272, 272, 272, 251, 268, 272, 259, 266, 258, 273]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835390753Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"[2]R generation[0..12]: [134, 145, 144, 144, 144, 133, 142, 144, 140, 142, 139, 145]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835401062Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.83541189Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Max found gen is 146","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835422739Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Generation requested: 146 >= found:146","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835432467Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Next flush: 274","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835441526Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835450635Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"":"downstairs","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835460133Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835468742Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"client":"0","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.83547835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"client":"1","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835487919Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"client":"2","":"downstairs","session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835497998Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:15:35.835507186Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"33769ff1-23d7-450d-9c08-29c3d1cd4fd3 is now active with session: e9e1d7e1-70af-4772-8161-903a982b4c27","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835518965Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835527923Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835535392Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721,"session_id":"e9e1d7e1-70af-4772-8161-903a982b4c27"} {"msg":"Activated sub_volume 33769ff1-23d7-450d-9c08-29c3d1cd4fd3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:35.835544591Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 509 len 20480 data: 12 12 11 11 11 02/30 Read block 51 len 12288 03/30 Write block 488 len 36864 data: 8 7 8 8 6 6 8 9 8 04/30 Flush 05/30 Flush 06/30 Flush 07/30 Write block 326 len 8192 data: 9 13 08/30 Read block 411 len 24576 09/30 Write block 70 len 40960 data: 10 12 13 11 11 10 11 8 10 8 10/30 Read block 17 len 8192 11/30 Read block 25 len 4096 12/30 Write block 434 len 40960 data: 21 21 20 18 20 15 13 14 15 13 13/30 Write block 191 len 36864 data: 12 14 15 16 16 14 14 14 12 14/30 Read block 86 len 20480 15/30 Write block 307 len 20480 data: 9 9 12 12 14 16/30 Read block 574 len 36864 17/30 Read block 236 len 8192 18/30 Read block 177 len 16384 19/30 Read block 166 len 28672 20/30 Write block 27 len 32768 data: 9 11 11 11 8 7 7 6 21/30 Flush 22/30 Write block 311 len 12288 data: 15 12 14 23/30 Read block 140 len 8192 24/30 Read block 61 len 12288 25/30 Write block 115 len 24576 data: 7 9 13 11 10 12 26/30 Read block 163 len 16384 27/30 Write block 233 len 28672 data: 9 7 6 6 8 12 12 28/30 Write block 32 len 24576 data: 8 8 7 8 8 12 29/30 Write block 221 len 20480 data: 7 10 14 12 13 30/30 Read block 210 len 32768 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Write 9 Done Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Read 6 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 2 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Write 5 Sent Done Done false 1029 Acked Read 8 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 1 0 28 Done 3 29 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:35.844996479Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1721} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 145 146 146 273 277 277 F T T <--- 2 040-059 144 144 144 272 272 272 F F F 3 060-079 144 146 146 272 277 277 F F F <--- 4 080-099 144 144 144 272 272 272 F F F 5 100-119 133 133 133 251 251 251 F T T <--- 6 120-139 142 142 142 268 268 268 F T T <--- 7 140-159 144 144 144 272 272 272 F F F 8 160-179 140 140 140 259 259 259 F F F 9 180-199 142 146 146 266 277 277 F F F <--- 10 200-219 139 139 139 258 258 258 F F F 11 220-239 145 145 145 273 273 273 F T T <--- 12 240-259 144 144 144 272 272 272 F F F 13 260-279 145 145 145 273 273 273 F F F 14 280-299 145 145 145 273 273 273 F F F 15 300-319 144 146 146 271 277 277 F T T <--- 16 320-339 140 146 146 263 277 277 F F F <--- 17 340-359 141 141 141 265 265 265 F F F 18 360-379 138 138 138 256 256 256 F F F 19 380-399 144 144 144 271 271 271 F F F 20 400-419 140 140 140 263 263 263 F F F 21 420-439 140 146 146 261 277 277 F F F <--- 22 440-459 142 146 146 268 277 277 F F F <--- 23 460-479 143 143 143 270 270 270 F F F 24 480-499 144 146 146 272 274 274 T F F <--- 25 500-519 143 146 146 270 274 274 T F F <--- 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 142 142 142 269 269 269 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 146, Max flush: 277 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 73 Verifying data now No change after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 147 147 147 278 278 278 F F F 2 040-059 144 144 144 272 272 272 F F F 3 060-079 146 146 146 277 277 277 F F F 4 080-099 144 144 144 272 272 272 F F F 5 100-119 147 147 147 278 278 278 F F F 6 120-139 147 147 147 278 278 278 F F F 7 140-159 144 144 144 272 272 272 F F F 8 160-179 140 140 140 259 259 259 F F F 9 180-199 146 146 146 277 277 277 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 147 147 147 278 278 278 F F F 12 240-259 144 144 144 272 272 272 F F F 13 260-279 145 145 145 273 273 273 F F F 14 280-299 145 145 145 273 273 273 F F F 15 300-319 147 147 147 278 278 278 F F F 16 320-339 146 146 146 277 277 277 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 138 138 138 256 256 256 F F F 19 380-399 144 144 144 271 271 271 F F F 20 400-419 140 140 140 263 263 263 F F F 21 420-439 146 146 146 277 277 277 F F F 22 440-459 146 146 146 277 277 277 F F F 23 460-479 143 143 143 270 270 270 F F F 24 480-499 146 146 146 274 274 274 F F F 25 500-519 146 146 146 274 274 274 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 142 142 142 269 269 269 F F F 29 580-599 123 123 123 229 229 229 F F F Max gen: 147, Max flush: 278 Begin loop 74 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.101980849Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.110248431Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.110541448Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.110567174Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.110575433Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.110586671Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729} {"msg":"Crucible 11c95449-3478-4816-9849-b65723dbe23d has session id: cd8a22f0-b29d-4217-9e70-fb64b65ff2a6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.110935131Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Upstairs opts: Upstairs UUID: 11c95449-3478-4816-9849-b65723dbe23d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.110963856Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Crucible stats registered with UUID: 11c95449-3478-4816-9849-b65723dbe23d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.110975225Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:40.110984543Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.11114105Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"io task","client":"0","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111169916Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"io task","client":"2","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111180205Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"io task","client":"1","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111308796Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"task":"crutest"} {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111325593Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729} {"msg":"11c95449-3478-4816-9849-b65723dbe23d active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.11134914Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111358439Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"client":"0","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111369817Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"client":"1","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111379326Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"client":"2","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111388864Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"io task","client":"2","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.11141926Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"io task","client":"0","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111439897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"io task","client":"1","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"ds_connection connected from Ok(127.0.0.1:54909)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111463253Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"io task","client":"1","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"ds_connection connected from Ok(127.0.0.1:56022)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111473722Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"io task","client":"0","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"ds_connection connected from Ok(127.0.0.1:44597)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.111500918Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"io task","client":"2","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112187687Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"client":"1","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112206495Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"client":"0","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112292562Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"client":"2","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112502621Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.11251274Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"[0]R flush_numbers[0..12]: [253, 278, 272, 277, 272, 278, 278, 272, 259, 277, 258, 278]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112603947Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"[0]R generation[0..12]: [134, 147, 144, 146, 144, 147, 147, 144, 140, 146, 139, 147]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112615095Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112628063Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"[1]R flush_numbers[0..12]: [253, 278, 272, 277, 272, 278, 278, 272, 259, 277, 258, 278]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112638702Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"[1]R generation[0..12]: [134, 147, 144, 146, 144, 147, 147, 144, 140, 146, 139, 147]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.11264904Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112659778Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"[2]R flush_numbers[0..12]: [253, 278, 272, 277, 272, 278, 278, 272, 259, 277, 258, 278]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112670287Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"[2]R generation[0..12]: [134, 147, 144, 146, 144, 147, 147, 144, 140, 146, 139, 147]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112680655Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112691414Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Max found gen is 148","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112700722Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Generation requested: 148 >= found:148","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112710261Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Next flush: 279","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.11271938Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112728638Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"":"downstairs","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112738267Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112746946Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"client":"0","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112756504Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"client":"1","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112765983Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"client":"2","":"downstairs","session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:15:40.112777121Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112790129Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"11c95449-3478-4816-9849-b65723dbe23d is now active with session: cd8a22f0-b29d-4217-9e70-fb64b65ff2a6","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112799988Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112808627Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729,"session_id":"cd8a22f0-b29d-4217-9e70-fb64b65ff2a6"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112817675Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729} {"msg":"Activated sub_volume 11c95449-3478-4816-9849-b65723dbe23d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:40.112825724Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 390 len 16384 data: 11 14 14 14 02/30 Read block 574 len 4096 03/30 Write block 312 len 12288 data: 13 15 12 04/30 Read block 390 len 12288 05/30 Write block 404 len 12288 data: 12 13 13 06/30 Read block 485 len 24576 07/30 Read block 73 len 8192 08/30 Write block 127 len 36864 data: 13 11 11 12 12 13 12 11 9 09/30 Read block 125 len 32768 10/30 Write block 164 len 20480 data: 21 20 17 16 13 11/30 Read block 361 len 12288 12/30 Write block 318 len 8192 data: 16 17 13/30 Write block 481 len 32768 data: 8 9 11 9 9 8 8 9 14/30 Read block 237 len 4096 15/30 Write block 391 len 24576 data: 15 15 15 10 10 9 16/30 Read block 14 len 32768 17/30 Read block 455 len 20480 18/30 Write block 576 len 36864 data: 15 13 13 12 10 9 9 10 11 19/30 Write block 144 len 28672 data: 15 15 12 10 9 9 8 20/30 Read block 272 len 4096 21/30 Read block 207 len 20480 22/30 Read block 396 len 24576 23/30 Write block 152 len 12288 data: 9 7 5 24/30 Read block 130 len 24576 25/30 Read block 247 len 24576 26/30 Write block 327 len 8192 data: 14 13 27/30 Write block 154 len 4096 data: 6 28/30 Write block 413 len 24576 data: 9 8 8 8 9 10 29/30 Write block 367 len 24576 data: 6 6 7 7 9 9 30/30 Read block 94 len 40960 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 1 Done Done Sent false 1002 Acked Write 3 Done Done Sent false 1003 Acked Read 3 Done Done Sent false 1004 Acked Write 3 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Read 2 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Write 5 Done Done Sent false 1010 Acked Read 3 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Read 8 Done Done Sent false 1016 Acked Read 5 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Read 6 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Write 1 Done Done Sent false 1027 Acked Write 6 Done Done Sent false 1028 Acked Write 6 Done Done Sent false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:40.121716445Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1729} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 147 147 147 278 278 278 F F F 2 040-059 144 144 144 272 272 272 F F F 3 060-079 146 146 146 277 277 277 F F F 4 080-099 144 144 144 272 272 272 F F F 5 100-119 147 147 147 278 278 278 F F F 6 120-139 147 147 147 278 278 278 T T F <--- 7 140-159 144 144 144 272 272 272 T T F <--- 8 160-179 140 140 140 259 259 259 T T F <--- 9 180-199 146 146 146 277 277 277 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 147 147 147 278 278 278 F F F 12 240-259 144 144 144 272 272 272 F F F 13 260-279 145 145 145 273 273 273 F F F 14 280-299 145 145 145 273 273 273 F F F 15 300-319 147 147 147 278 278 278 T T F <--- 16 320-339 146 146 146 277 277 277 T T F <--- 17 340-359 141 141 141 265 265 265 F F F 18 360-379 138 138 138 256 256 256 T T F <--- 19 380-399 144 144 144 271 271 271 T T T 20 400-419 140 140 140 263 263 263 T T F <--- 21 420-439 146 146 146 277 277 277 F F F 22 440-459 146 146 146 277 277 277 F F F 23 460-479 143 143 143 270 270 270 F F F 24 480-499 146 146 146 274 274 274 T T F <--- 25 500-519 146 146 146 274 274 274 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 142 142 142 269 269 269 T T F <--- 29 580-599 123 123 123 229 229 229 T T F <--- Max gen: 147, Max flush: 278 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 147 147 147 278 278 278 F F F 2 040-059 144 144 144 272 272 272 F F F 3 060-079 146 146 146 277 277 277 F F F 4 080-099 144 144 144 272 272 272 F F F 5 100-119 147 147 147 278 278 278 F F F 6 120-139 148 148 148 279 279 279 F F F 7 140-159 148 148 148 279 279 279 F F F 8 160-179 148 148 148 279 279 279 F F F 9 180-199 146 146 146 277 277 277 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 147 147 147 278 278 278 F F F 12 240-259 144 144 144 272 272 272 F F F 13 260-279 145 145 145 273 273 273 F F F 14 280-299 145 145 145 273 273 273 F F F 15 300-319 148 148 148 279 279 279 F F F 16 320-339 148 148 148 279 279 279 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 148 148 148 279 279 279 F F F 19 380-399 148 148 148 279 279 279 F F F 20 400-419 148 148 148 279 279 279 F F F 21 420-439 146 146 146 277 277 277 F F F 22 440-459 146 146 146 277 277 277 F F F 23 460-479 143 143 143 270 270 270 F F F 24 480-499 148 148 148 279 279 279 F F F 25 500-519 146 146 146 274 274 274 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 148 148 148 279 279 279 F F F 29 580-599 148 148 148 279 279 279 F F F Max gen: 148, Max flush: 279 Begin loop 75 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.376999634Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:44.384239786Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.384303487Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.391233204Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.391505484Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.39153327Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.391542139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.391553527Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737} {"msg":"Crucible 7eb6d121-52fd-4e0d-94b2-fe5cab3d8a4c has session id: 8c325aa1-42f6-4d27-a6e2-3c22d18a1254","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.391894047Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Upstairs opts: Upstairs UUID: 7eb6d121-52fd-4e0d-94b2-fe5cab3d8a4c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.391916014Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Crucible stats registered with UUID: 7eb6d121-52fd-4e0d-94b2-fe5cab3d8a4c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.391926133Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:44.391934991Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392096048Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"io task","client":"1","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392134872Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"io task","client":"0","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.39214581Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"io task","client":"2","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392249365Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"task":"crutest"} {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392266323Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737} {"msg":"7eb6d121-52fd-4e0d-94b2-fe5cab3d8a4c active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392306837Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392316226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"client":"0","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392326004Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"client":"1","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392335513Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"client":"2","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392345111Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"io task","client":"2","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.39235503Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"io task","client":"1","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392364728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"io task","client":"0","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"ds_connection connected from Ok(127.0.0.1:59499)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392412821Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"io task","client":"2","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"ds_connection connected from Ok(127.0.0.1:57710)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392440157Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"io task","client":"1","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"ds_connection connected from Ok(127.0.0.1:41471)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.392451616Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"io task","client":"0","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393135416Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"client":"2","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393155383Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"client":"1","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393238911Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"client":"0","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393443111Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393452759Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"[0]R flush_numbers[0..12]: [253, 278, 272, 277, 272, 278, 279, 279, 279, 277, 258, 278]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393524239Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"[0]R generation[0..12]: [134, 147, 144, 146, 144, 147, 148, 148, 148, 146, 139, 147]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393535187Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393546126Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"[1]R flush_numbers[0..12]: [253, 278, 272, 277, 272, 278, 279, 279, 279, 277, 258, 278]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393556594Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"[1]R generation[0..12]: [134, 147, 144, 146, 144, 147, 148, 148, 148, 146, 139, 147]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393567033Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393577971Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"[2]R flush_numbers[0..12]: [253, 278, 272, 277, 272, 278, 279, 279, 279, 277, 258, 278]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393590579Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"[2]R generation[0..12]: [134, 147, 144, 146, 144, 147, 148, 148, 148, 146, 139, 147]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393601148Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393611976Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Max found gen is 149","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393621325Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Generation requested: 150 >= found:149","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393630933Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Next flush: 280","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393640252Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393649551Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"":"downstairs","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393659249Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393667928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"client":"0","":"downstairs","session_id"Wait for a query_work_queue command to finish before sending IO :"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393682136Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"client":"1","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393691615Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"client":"2","":"downstairs","session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393701253Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393710332Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"7eb6d121-52fd-4e0d-94b2-fe5cab3d8a4c is now active with session: 8c325aa1-42f6-4d27-a6e2-3c22d18a1254","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.39372007Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393729359Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393736928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737,"session_id":"8c325aa1-42f6-4d27-a6e2-3c22d18a1254"} {"msg":"Activated sub_volume 7eb6d121-52fd-4e0d-94b2-fe5cab3d8a4c","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:44.393757635Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 25 len 28672 data: 10 10 10 12 12 12 9 02/30 Write block 516 len 4096 data: 10 03/30 Write block 186 len 20480 data: 10 9 8 8 11 04/30 Read block 556 len 40960 05/30 Write block 319 len 16384 data: 18 15 9 8 06/30 Read block 170 len 24576 07/30 Read block 127 len 8192 08/30 Flush 09/30 Flush 10/30 Flush 11/30 Write block 502 len 20480 data: 7 5 4 8 7 12/30 Read block 459 len 8192 13/30 Read block 49 len 24576 14/30 Write block 79 len 20480 data: 9 8 13 14 13 15/30 Read block 589 len 20480 16/30 Write block 81 len 8192 data: 14 15 17/30 Write block 596 len 4096 data: 8 18/30 Write block 374 len 28672 data: 12 11 14 15 12 14 12 19/30 Write block 289 len 12288 data: 11 12 12 20/30 Write block 265 len 32768 data: 16 17 17 15 15 12 13 12 21/30 Read block 32 len 4096 22/30 Write block 37 len 20480 data: 13 14 14 15 16 23/30 Write block 65 len 32768 data: 15 12 12 11 12 11 13 14 24/30 Flush 25/30 Write block 249 len 28672 data: 11 12 10 8 5 6 7 26/30 Read block 81 len 12288 27/30 Read block 300 len 4096 28/30 Read block 439 len 12288 29/30 Write block 52 len 40960 data: 11 10 12 12 15 14 12 11 9 11 30/30 Write block 221 len 16384 data: 8 11 15 13 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Read 10 Done Done Done false 1004 Acked Write 4 Done Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Read 6 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 2 2 29 Done 5 28 28 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:44.401823346Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1737} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 147 150 150 278 283 283 T F F <--- 2 040-059 144 150 150 272 283 283 F T T <--- 3 060-079 146 150 150 277 283 283 F T T <--- 4 080-099 144 150 150 272 283 283 F F F <--- 5 100-119 147 147 147 278 278 278 F F F 6 120-139 148 148 148 279 279 279 F F F 7 140-159 148 148 148 279 279 279 F F F 8 160-179 148 148 148 279 279 279 F F F 9 180-199 146 150 150 277 280 280 T F F <--- 10 200-219 139 139 139 258 258 258 F F F 11 220-239 147 147 147 278 278 278 F T T <--- 12 240-259 144 144 144 272 272 272 F T T <--- 13 260-279 145 150 150 273 283 283 F F F <--- 14 280-299 145 150 150 273 283 283 F F F <--- 15 300-319 148 150 150 279 280 280 T F F <--- 16 320-339 148 150 150 279 280 280 T F F <--- 17 340-359 141 141 141 265 265 265 F F F 18 360-379 148 150 150 279 283 283 F F F <--- 19 380-399 148 150 150 279 283 283 F F F <--- 20 400-419 148 148 148 279 279 279 F F F 21 420-439 146 146 146 277 277 277 F F F 22 440-459 146 146 146 277 277 277 F F F 23 460-479 143 143 143 270 270 270 F F F 24 480-499 148 148 148 279 279 279 F F F 25 500-519 146 150 150 274 283 283 T F F <--- 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 148 148 148 279 279 279 F F F 29 580-599 148 150 150 279 283 283 F F F <--- Max gen: 150, Max flush: 283 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 150 150 150 283 283 283 F F F 2 040-059 151 151 151 284 284 284 F F F 3 060-079 151 151 151 284 284 284 F F F 4 080-099 150 150 150 283 283 283 F F F 5 100-119 147 147 147 278 278 278 F F F 6 120-139 148 148 148 279 279 279 F F F 7 140-159 148 148 148 279 279 279 F F F 8 160-179 148 148 148 279 279 279 F F F 9 180-199 150 150 150 280 280 280 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 151 151 151 284 284 284 F F F 12 240-259 151 151 151 284 284 284 F F F 13 260-279 150 150 150 283 283 283 F F F 14 280-299 150 150 150 283 283 283 F F F 15 300-319 150 150 150 280 280 280 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 150 150 150 283 283 283 F F F 19 380-399 150 150 150 283 283 283 F F F 20 400-419 148 148 148 279 279 279 F F F 21 420-439 146 146 146 277 277 277 F F F 22 440-459 146 146 146 277 277 277 F F F 23 460-479 143 143 143 270 270 270 F F F 24 480-499 148 148 148 279 279 279 F F F 25 500-519 150 150 150 283 283 283 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 148 148 148 279 279 279 F F F 29 580-599 150 150 150 283 283 283 F F F Max gen: 151, Max flush: 284 Begin loop 76 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.662808662Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.671078183Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.671368711Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.671395807Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.671404476Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.671415664Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745} {"msg":"Crucible 497b2ed8-7d96-4d78-baba-866c4e5e10ce has session id: 4e34b640-d135-4077-ac75-e19c3a4acba1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.671740626Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Upstairs opts: Upstairs UUID: 497b2ed8-7d96-4d78-baba-866c4e5e10ce, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.671774551Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Crucible stats registered with UUID: 497b2ed8-7d96-4d78-baba-866c4e5e10ce","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.67178578Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:48.671795418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.671952235Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"io task","client":"2","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.67198805Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"io task","client":"0","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.671999139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"io task","client":"1","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672116551Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"task":"crutest"} {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672133949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745} {"msg":"497b2ed8-7d96-4d78-baba-866c4e5e10ce active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672170074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672179432Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"client":"0","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672189071Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"client":"1","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672198529Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"client":"2","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672207868Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"io task","client":"2","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672217807Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"io task","client":"0","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672227715Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"io task","client":"1","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"ds_connection connected from Ok(127.0.0.1:62016)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672237864Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"io task","client":"2","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"ds_connection connected from Ok(127.0.0.1:32769)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672406979Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"io task","client":"0","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"ds_connection connected from Ok(127.0.0.1:44415)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672418497Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"io task","client":"1","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.672958748Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"client":"2","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673008311Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"client":"1","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673058754Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"client":"0","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673283991Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673299409Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"[0]R flush_numbers[0..12]: [253, 283, 284, 284, 283, 278, 279, 279, 279, 280, 258, 284]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673353011Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"[0]R generation[0..12]: [134, 150, 151, 151, 150, 147, 148, 148, 148, 150, 139, 151]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673364179Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673374988Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"[1]R flush_numbers[0..12]: [253, 283, 284, 284, 283, 278, 279, 279, 279, 280, 258, 284]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673385436Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"[1]R generation[0..12]: [134, 150, 151, 151, 150, 147, 148, 148, 148, 150, 139, 151]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673395854Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673406543Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"[2]R flush_numbers[0..12]: [253, 283, 284, 284, 283, 278, 279, 279, 279, 280, 258, 284]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673416981Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"[2]R generation[0..12]: [134, 150, 151, 151, 150, 147, 148, 148, 148, 150, 139, 151]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.6734272Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673441748Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673451016Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673460575Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Next flush: 285","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673469724Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673478902Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"":"downstairs","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673489951Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673498609Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"client":"0","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673508098Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"client":"1","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673517427Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"client":"2","":"downstairs","session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673527085Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673535974Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"497b2ed8-7d96-4d78-baba-866c4e5e10ce is now active with session: 4e34b640-d135-4077-ac75-e19c3a4acba1","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673545803Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Wait for a query_work_queue command to finish before sending IO The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673559041Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.67356656Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745,"session_id":"4e34b640-d135-4077-ac75-e19c3a4acba1"} {"msg":"Activated sub_volume 497b2ed8-7d96-4d78-baba-866c4e5e10ce","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:48.673575898Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 396 len 32768 data: 10 9 11 10 10 10 12 11 02/30 Read block 263 len 8192 03/30 Read block 25 len 32768 04/30 Read block 372 len 24576 05/30 Flush 06/30 Write block 150 len 32768 data: 9 10 10 8 7 5 4 8 07/30 Read block 468 len 24576 08/30 Write block 82 len 4096 data: 16 09/30 Write block 472 len 24576 data: 16 18 17 14 9 7 10/30 Flush 11/30 Write block 170 len 8192 data: 10 11 12/30 Read block 387 len 28672 13/30 Write block 460 len 12288 data: 15 13 11 14/30 Read block 497 len 32768 15/30 Flush 16/30 Write block 47 len 4096 data: 14 17/30 Write block 123 len 36864 data: 10 12 11 13 14 12 12 13 13 18/30 Read block 186 len 36864 19/30 Read block 424 len 8192 20/30 Read block 534 len 4096 21/30 Write block 510 len 4096 data: 13 22/30 Write block 294 len 32768 data: 12 8 8 7 8 7 5 6 23/30 Read block 492 len 12288 24/30 Write block 109 len 24576 data: 13 13 14 11 9 11 25/30 Write block 586 len 16384 data: 9 12 11 8 26/30 Read block 599 len 4096 27/30 Read block 130 len 20480 28/30 Read block 242 len 4096 29/30 Read block 527 len 8192 30/30 Write block 428 len 40960 data: 11 11 11 14 16 19 22 22 21 19 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:20 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1010 Acked Write 2 Done Done Done false 1011 Acked Read 7 Done Done Done false 1012 Acked Write 3 Done Done Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Write 6 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 17 1 19 Done 19 3 19 41 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1009)) Some(JobId(1014)) Downstairs last five completed: 1009 1008 1007 1006 1005 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:48.682330289Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1745} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 150 150 150 283 283 283 F F F 2 040-059 151 151 151 284 284 284 T F T <--- 3 060-079 151 151 151 284 284 284 F F F 4 080-099 152 152 152 286 286 286 F F F 5 100-119 147 147 147 278 278 278 T F T <--- 6 120-139 148 148 148 279 279 279 T F T <--- 7 140-159 152 152 152 286 286 286 F F F 8 160-179 152 148 152 287 279 287 F T F <--- 9 180-199 150 150 150 280 280 280 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 151 151 151 284 284 284 F F F 12 240-259 151 151 151 284 284 284 F F F 13 260-279 150 150 150 283 283 283 F F F 14 280-299 150 150 150 283 283 283 T F T <--- 15 300-319 150 150 150 280 280 280 T F T <--- 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 150 150 150 283 283 283 F F F 19 380-399 152 152 152 285 285 285 F F F 20 400-419 152 152 152 285 285 285 F F F 21 420-439 146 146 146 277 277 277 T F T <--- 22 440-459 146 146 146 277 277 277 F F F 23 460-479 152 152 152 287 286 287 F T F <--- 24 480-499 148 148 148 279 279 279 F F F 25 500-519 150 150 150 283 283 283 T F T <--- 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 148 148 148 279 279 279 F F F 29 580-599 150 150 150 283 283 283 T F T <--- Max gen: 152, Max flush: 287 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 150 150 150 283 283 283 F F F 2 040-059 153 153 153 288 288 288 F F F 3 060-079 151 151 151 284 284 284 F F F 4 080-099 152 152 152 286 286 286 F F F 5 100-119 153 153 153 288 288 288 F F F 6 120-139 153 153 153 288 288 288 F F F 7 140-159 152 152 152 286 286 286 F F F 8 160-179 152 152 152 287 287 287 F F F 9 180-199 150 150 150 280 280 280 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 151 151 151 284 284 284 F F F 12 240-259 151 151 151 284 284 284 F F F 13 260-279 150 150 150 283 283 283 F F F 14 280-299 153 153 153 288 288 288 F F F 15 300-319 153 153 153 288 288 288 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 150 150 150 283 283 283 F F F 19 380-399 152 152 152 285 285 285 F F F 20 400-419 152 152 152 285 285 285 F F F 21 420-439 153 153 153 288 288 288 F F F 22 440-459 146 146 146 277 277 277 F F F 23 460-479 152 152 152 287 287 287 F F F 24 480-499 148 148 148 279 279 279 F F F 25 500-519 153 153 153 288 288 288 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 F F F 28 560-579 148 148 148 279 279 279 F F F 29 580-599 153 153 153 288 288 288 F F F Max gen: 153, Max flush: 288 Begin loop 77 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.939747196Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.947944068Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.948244754Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.948276559Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.948285278Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.948296746Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753} {"msg":"Crucible ad55f8ee-e033-42bd-b6c6-e501ecf19a2f has session id: 9a008b0e-24c0-408f-a048-5f6a71ec6d46","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.948674641Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Upstairs opts: Upstairs UUID: ad55f8ee-e033-42bd-b6c6-e501ecf19a2f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.948700407Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Crucible stats registered with UUID: ad55f8ee-e033-42bd-b6c6-e501ecf19a2f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.948710266Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:52.948719174Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.948879071Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"io task","client":"0","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.948937372Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"io task","client":"2","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.94895109Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"io task","client":"1","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949077902Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"task":"crutest"} {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949095709Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753} {"msg":"ad55f8ee-e033-42bd-b6c6-e501ecf19a2f active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949127865Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949137033Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"client":"0","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949146692Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"client":"1","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.94915612Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"client":"2","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949165559Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"io task","client":"2","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949175398Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"io task","client":"0","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949185296Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"io task","client":"1","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"ds_connection connected from Ok(127.0.0.1:54234)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949275213Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"io task","client":"0","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"ds_connection connected from Ok(127.0.0.1:43466)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949334204Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"io task","client":"2","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"ds_connection connected from Ok(127.0.0.1:48082)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949357631Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"io task","client":"1","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949939266Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"client":"0","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.949985759Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"client":"2","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950006696Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"client":"1","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950272167Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950283336Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"[0]R flush_numbers[0..12]: [253, 283, 288, 284, 286, 288, 288, 286, 287, 280, 258, 284]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950308902Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"[0]R generation[0..12]: [134, 150, 153, 151, 152, 153, 153, 152, 152, 150, 139, 151]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.95032039Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950331559Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"[1]R flush_numbers[0..12]: [253, 283, 288, 284, 286, 288, 288, 286, 287, 280, 258, 284]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950342207Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"[1]R generation[0..12]: [134, 150, 153, 151, 152, 153, 153, 152, 152, 150, 139, 151]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950352666Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950363514Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"[2]R flush_numbers[0..12]: [253, 283, 288, 284, 286, 288, 288, 286, 287, 280, 258, 284]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950374092Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"[2]R generation[0..12]: [134, 150, 153, 151, 152, 153, 153, 152, 152, 150, 139, 151]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950386441Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950397199Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Max found gen is 154","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950406628Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Generation requested: 154 >= found:154","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950416366Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Next flush: 289","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950425565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950434884Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"":"downstairs","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950444442Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950453101Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"client":"0","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950462699Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"client":"1","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950472238Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"client":"2","":"downstairs","session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950481967Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950490955Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"ad55f8ee-e033-42bd-b6c6-e501ecf19a2f is now active with session: 9a008b0e-24c0-408f-a048-5f6a71ec6d46","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950500974Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2026-04-15T18:15:52.950514672Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950522361Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753,"session_id":"9a008b0e-24c0-408f-a048-5f6a71ec6d46"} {"msg":"Activated sub_volume ad55f8ee-e033-42bd-b6c6-e501ecf19a2f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:52.950531339Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 266 len 32768 data: 18 18 16 16 13 14 13 13 02/30 Read block 300 len 8192 03/30 Flush 04/30 Write block 311 len 8192 data: 16 14 05/30 Write block 181 len 12288 data: 16 16 16 06/30 Read block 581 len 24576 07/30 Read block 178 len 28672 08/30 Write block 37 len 32768 data: 14 15 15 16 17 16 17 14 09/30 Read block 587 len 40960 10/30 Read block 411 len 4096 11/30 Read block 201 len 32768 12/30 Read block 458 len 20480 13/30 Read block 291 len 24576 14/30 Read block 87 len 28672 15/30 Read block 14 len 28672 16/30 Write block 51 len 28672 data: 10 12 11 13 13 16 15 17/30 Flush 18/30 Write block 483 len 4096 data: 12 19/30 Read block 547 len 32768 20/30 Write block 184 len 20480 data: 12 9 11 10 9 21/30 Write block 181 len 20480 data: 17 17 17 13 10 22/30 Read block 41 len 4096 23/30 Write block 552 len 20480 data: 9 11 12 12 13 24/30 Write block 40 len 20480 data: 17 18 17 18 15 25/30 Read block 510 len 8192 26/30 Read block 590 len 32768 27/30 Read block 586 len 20480 28/30 Write block 239 len 36864 data: 13 12 15 16 16 16 15 16 14 29/30 Write block 36 len 28672 data: 9 15 16 16 18 19 18 30/30 Write block 269 len 32768 data: 17 14 15 14 14 11 7 7 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 2 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Read 7 Done Sent Done false 1007 Acked Write 8 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Write 5 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 9 Done Sent Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 28 3 33 Done 28 2 27 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) None Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:52.959162548Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1753} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 154 150 154 290 283 290 T F T <--- 2 040-059 154 153 154 290 288 290 T F T <--- 3 060-079 151 151 151 284 284 284 F F F 4 080-099 152 152 152 286 286 286 F F F 5 100-119 153 153 153 288 288 288 F F F 6 120-139 153 153 153 288 288 288 F F F 7 140-159 152 152 152 286 286 286 F F F 8 160-179 152 152 152 287 287 287 F F F 9 180-199 154 150 154 290 280 290 T F T <--- 10 200-219 139 139 139 258 258 258 F F F 11 220-239 151 151 151 284 284 284 T F T <--- 12 240-259 151 151 151 284 284 284 T F T <--- 13 260-279 154 150 154 289 283 289 T T T <--- 14 280-299 153 153 153 288 288 288 F F F 15 300-319 154 153 154 290 288 290 F F F <--- 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 150 150 150 283 283 283 F F F 19 380-399 152 152 152 285 285 285 F F F 20 400-419 152 152 152 285 285 285 F F F 21 420-439 153 153 153 288 288 288 F F F 22 440-459 146 146 146 277 277 277 F F F 23 460-479 152 152 152 287 287 287 F F F 24 480-499 148 148 148 279 279 279 T F T <--- 25 500-519 153 153 153 288 288 288 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 140 140 140 259 259 259 T F T <--- 28 560-579 148 148 148 279 279 279 F F F 29 580-599 153 153 153 288 288 288 F F F Max gen: 154, Max flush: 290 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 77 Verifying data now No change after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 253 253 253 F F F 1 020-039 155 155 155 291 291 291 F F F 2 040-059 155 155 155 291 291 291 F F F 3 060-079 151 151 151 284 284 284 F F F 4 080-099 152 152 152 286 286 286 F F F 5 100-119 153 153 153 288 288 288 F F F 6 120-139 153 153 153 288 288 288 F F F 7 140-159 152 152 152 286 286 286 F F F 8 160-179 152 152 152 287 287 287 F F F 9 180-199 155 155 155 291 291 291 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 155 155 155 291 291 291 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 155 155 155 291 291 291 F F F 14 280-299 153 153 153 288 288 288 F F F 15 300-319 154 154 154 290 290 290 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 150 150 150 283 283 283 F F F 19 380-399 152 152 152 285 285 285 F F F 20 400-419 152 152 152 285 285 285 F F F 21 420-439 153 153 153 288 288 288 F F F 22 440-459 146 146 146 277 277 277 F F F 23 460-479 152 152 152 287 287 287 F F F 24 480-499 155 155 155 291 291 291 F F F 25 500-519 153 153 153 288 288 288 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 155 155 155 291 291 291 F F F 28 560-579 148 148 148 279 279 279 F F F 29 580-599 153 153 153 288 288 288 F F F Max gen: 155, Max flush: 291 Begin loop 78 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.206237927Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.21435763Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.214648758Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.214676114Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.214684612Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.214696001Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761} {"msg":"Crucible 7c16ebf7-67fd-4815-9d78-e0a0c03b7f99 has session id: f66f38a5-ad47-43fe-ac2f-b31df7f04581","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215069166Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Upstairs opts: Upstairs UUID: 7c16ebf7-67fd-4815-9d78-e0a0c03b7f99, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215097792Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Crucible stats registered with UUID: 7c16ebf7-67fd-4815-9d78-e0a0c03b7f99","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.21510891Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:57.215118569Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215279485Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"io task","client":"1","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215335767Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"io task","client":"0","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215347156Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"io task","client":"2","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215484265Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"task":"crutest"} {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215539487Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761} {"msg":"7c16ebf7-67fd-4815-9d78-e0a0c03b7f99 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215562204Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215572323Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"client":"0","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215582761Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"client":"1","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.21559247Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"client":"2","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215602268Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"io task","client":"2","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215612097Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"io task","client":"0","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215621925Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"io task","client":"1","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"ds_connection connected from Ok(127.0.0.1:55633)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215762565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"io task","client":"2","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"ds_connection connected from Ok(127.0.0.1:38365)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215774313Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"io task","client":"0","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"ds_connection connected from Ok(127.0.0.1:65205)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.215784542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"io task","client":"1","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216384344Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"client":"2","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216463702Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"client":"1","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216494788Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"client":"0","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216723864Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216744141Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"[0]R flush_numbers[0..12]: [253, 291, 291, 284, 286, 288, 288, 286, 287, 291, 258, 291]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216785615Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"[0]R generation[0..12]: [134, 155, 155, 151, 152, 153, 153, 152, 152, 155, 139, 155]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216797464Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216808552Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"[1]R flush_numbers[0..12]: [253, 291, 291, 284, 286, 288, 288, 286, 287, 291, 258, 291]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.21681901Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"[1]R generation[0..12]: [134, 155, 155, 151, 152, 153, 153, 152, 152, 155, 139, 155]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216829669Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216840397Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"[2]R flush_numbers[0..12]: [253, 291, 291, 284, 286, 288, 288, 286, 287, 291, 258, 291]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216850926Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"[2]R generation[0..12]: [134, 155, 155, 151, 152, 153, 153, 152, 152, 155, 139, 155]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216861244Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216872123Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216881541Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Generation requested: 156 >= found:156","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.21689121Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Next flush: 292","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216900488Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216909847Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"":"downstairs","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216919596Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Transition from WaitQuorum to Active","Wait for a query_work_queue command to finish before sending IO v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216933894Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"client":"0","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216943592Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"client":"1","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216953291Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"client":"2","":"downstairs","session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216963029Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216972068Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"7c16ebf7-67fd-4815-9d78-e0a0c03b7f99 is now active with session: f66f38a5-ad47-43fe-ac2f-b31df7f04581","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216981827Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216990925Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.216998394Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761,"session_id":"f66f38a5-ad47-43fe-ac2f-b31df7f04581"} {"msg":"Activated sub_volume 7c16ebf7-67fd-4815-9d78-e0a0c03b7f99","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:15:57.217007393Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 291 len 16384 data: 13 12 14 13 02/30 Write block 169 len 32768 data: 12 11 12 11 10 10 10 14 03/30 Flush 04/30 Write block 168 len 32768 data: 14 13 12 13 12 11 11 11 05/30 Read block 121 len 4096 06/30 Read block 525 len 40960 07/30 Read block 520 len 24576 08/30 Flush 09/30 Write block 366 len 16384 data: 9 7 7 8 10/30 Read block 524 len 32768 11/30 Write block 156 len 28672 data: 5 9 9 11 10 9 10 12/30 Write block 368 len 32768 data: 8 9 8 10 10 11 13 12 13/30 Write block 177 len 40960 data: 15 17 16 19 18 18 18 14 11 12 14/30 Read block 494 len 28672 15/30 Read block 461 len 24576 16/30 Write block 86 len 20480 data: 12 15 15 16 13 17/30 Write block 442 len 24576 data: 16 14 14 16 15 13 18/30 Write block 559 len 4096 data: 12 19/30 Read block 274 len 16384 20/30 Read block 3 len 12288 21/30 Write block 505 len 32768 data: 9 8 7 10 13 14 12 12 22/30 Write block 541 len 40960 data: 11 11 10 8 7 8 6 8 9 9 23/30 Read block 41 len 20480 24/30 Write block 17 len 40960 data: 13 12 10 10 9 7 7 7 11 11 25/30 Write block 77 len 40960 data: 9 11 10 9 15 17 14 13 15 13 26/30 Flush 27/30 Write block 229 len 36864 data: 16 16 13 10 10 8 7 7 9 28/30 Read block 13 len 40960 29/30 Read block 279 len 4096 30/30 Read block 478 len 20480 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 8 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Read 6 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Read 3 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 1 28 Done 27 0 26 53 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1002)) Some(JobId(1025)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:15:57.226480808Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1761} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 134 156 294 253 294 F F F <--- 1 020-039 156 155 156 294 291 294 F F F <--- 2 040-059 155 155 155 291 291 291 F F F 3 060-079 156 151 156 294 284 294 F F F <--- 4 080-099 156 152 156 294 286 294 F F F <--- 5 100-119 153 153 153 288 288 288 F F F 6 120-139 153 153 153 288 288 288 F F F 7 140-159 156 152 156 294 286 294 F F F <--- 8 160-179 156 156 156 294 292 294 F F F <--- 9 180-199 156 155 156 294 291 294 F F F <--- 10 200-219 139 139 139 258 258 258 F F F 11 220-239 155 155 155 291 291 291 T F T <--- 12 240-259 155 155 155 291 291 291 F F F 13 260-279 155 155 155 291 291 291 F F F 14 280-299 156 156 156 292 292 292 F F F 15 300-319 154 154 154 290 290 290 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 156 150 156 294 283 294 F F F <--- 19 380-399 152 152 152 285 285 285 F F F 20 400-419 152 152 152 285 285 285 F F F 21 420-439 153 153 153 288 288 288 F F F 22 440-459 156 146 156 294 277 294 F F F <--- 23 460-479 152 152 152 287 287 287 F F F 24 480-499 155 155 155 291 291 291 F F F 25 500-519 156 153 156 294 288 294 F F F <--- 26 520-539 143 143 143 270 270 270 F F F 27 540-559 156 155 156 294 291 294 F F F <--- 28 560-579 148 148 148 279 279 279 F F F 29 580-599 153 153 153 288 288 288 F F F Max gen: 156, Max flush: 294 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 294 294 294 F F F 1 020-039 156 156 156 294 294 294 F F F 2 040-059 155 155 155 291 291 291 F F F 3 060-079 156 156 156 294 294 294 F F F 4 080-099 156 156 156 294 294 294 F F F 5 100-119 153 153 153 288 288 288 F F F 6 120-139 153 153 153 288 288 288 F F F 7 140-159 156 156 156 294 294 294 F F F 8 160-179 156 156 156 294 294 294 F F F 9 180-199 156 156 156 294 294 294 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 155 155 155 291 291 291 F F F 14 280-299 156 156 156 292 292 292 F F F 15 300-319 154 154 154 290 290 290 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 156 156 156 294 294 294 F F F 19 380-399 152 152 152 285 285 285 F F F 20 400-419 152 152 152 285 285 285 F F F 21 420-439 153 153 153 288 288 288 F F F 22 440-459 156 156 156 294 294 294 F F F 23 460-479 152 152 152 287 287 287 F F F 24 480-499 155 155 155 291 291 291 F F F 25 500-519 156 156 156 294 294 294 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 156 156 156 294 294 294 F F F 28 560-579 148 148 148 279 279 279 F F F 29 580-599 153 153 153 288 288 288 F F F Max gen: 157, Max flush: 295 Begin loop 79 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.477580069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:01.484855496Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.484918577Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.4921444Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.492427969Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.492463184Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.492474502Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.492485881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769} {"msg":"Crucible 5b027319-5a2c-4781-989a-e67a2139e867 has session id: c2fc69ce-99d6-4701-8b9f-b3ecba6a415e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.492768519Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Upstairs opts: Upstairs UUID: 5b027319-5a2c-4781-989a-e67a2139e867, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.492784057Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Crucible stats registered with UUID: 5b027319-5a2c-4781-989a-e67a2139e867","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.492793776Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:01.492802764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.49297155Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"io task","client":"0","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493008834Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"io task","client":"2","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493020002Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"io task","client":"1","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.49310494Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"task":"crutest"} {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493129476Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769} {"msg":"5b027319-5a2c-4781-989a-e67a2139e867 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493146334Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493155223Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"client":"0","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493164801Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"client":"1","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.49317434Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"client":"2","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493183898Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"io task","client":"2","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493195887Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"io task","client":"0","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493205805Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"io task","client":"1","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"ds_connection connected from Ok(127.0.0.1:52261)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.49324515Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"io task","client":"2","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"ds_connection connected from Ok(127.0.0.1:65077)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493361683Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"io task","client":"0","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"ds_connection connected from Ok(127.0.0.1:58470)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493372881Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"io task","client":"1","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493871508Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"client":"2","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.493916181Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"client":"1","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494009398Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"client":"0","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.49420034Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494209949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"[0]R flush_numbers[0..12]: [294, 294, 291, 294, 294, 288, 288, 294, 294, 294, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494306944Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"[0]R generation[0..12]: [156, 156, 155, 156, 156, 153, 153, 156, 156, 156, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494318103Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494328971Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"[1]R flush_numbers[0..12]: [294, 294, 291, 294, 294, 288, 288, 294, 294, 294, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.49433959Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"[1]R generation[0..12]: [156, 156, 155, 156, 156, 153, 153, 156, 156, 156, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494352208Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494362886Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"[2]R flush_numbers[0..12]: [294, 294, 291, 294, 294, 288, 288, 294, 294, 294, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494373335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"[2]R generation[0..12]: [156, 156, 155, 156, 156, 153, 153, 156, 156, 156, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494383803Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494394592Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Max found gen is 158","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.49440386Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494413459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Next flush: 296","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494422617Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494431846Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"":"downstairs","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494441685Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494450343Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"client":"0","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494459852Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"client":"1","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494469641Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"client":"2","":"downstairs","session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"All required reconciliation work is completed","v":Wait for a query_work_queue command to finish before sending IO 0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494483918Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494493157Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"5b027319-5a2c-4781-989a-e67a2139e867 is now active with session: c2fc69ce-99d6-4701-8b9f-b3ecba6a415e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494504186Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494513124Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494520633Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769,"session_id":"c2fc69ce-99d6-4701-8b9f-b3ecba6a415e"} {"msg":"Activated sub_volume 5b027319-5a2c-4781-989a-e67a2139e867","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:01.494529702Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 508 len 16384 data: 11 14 15 13 02/30 Write block 408 len 12288 data: 9 8 9 03/30 Read block 123 len 32768 04/30 Read block 75 len 24576 05/30 Read block 272 len 40960 06/30 Write block 432 len 4096 data: 17 07/30 Flush 08/30 Write block 541 len 8192 data: 12 12 09/30 Flush 10/30 Write block 494 len 24576 data: 9 10 9 12 11 12 11/30 Write block 422 len 12288 data: 13 13 11 12/30 Write block 555 len 32768 data: 13 14 15 14 13 13 13 11 13/30 Read block 593 len 20480 14/30 Flush 15/30 Write block 365 len 8192 data: 9 10 16/30 Read block 124 len 16384 17/30 Write block 565 len 20480 data: 10 8 9 7 10 18/30 Read block 19 len 28672 19/30 Read block 567 len 20480 20/30 Read block 499 len 8192 21/30 Write block 308 len 20480 data: 10 13 13 17 15 22/30 Write block 283 len 28672 data: 12 12 13 12 10 12 12 23/30 Flush 24/30 Write block 453 len 12288 data: 9 12 14 25/30 Read block 118 len 16384 26/30 Write block 291 len 20480 data: 14 13 15 14 9 27/30 Read block 395 len 24576 28/30 Write block 458 len 24576 data: 18 17 16 14 12 11 29/30 Read block 335 len 28672 30/30 Write block 24 len 20480 data: 8 12 12 11 13 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 3 Sent Done Done false 1002 Acked Read 8 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Write 5 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Write 5 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:01.503475734Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1769} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 294 294 294 F F F 1 020-039 156 156 156 294 294 294 F T T <--- 2 040-059 155 155 155 291 291 291 F F F 3 060-079 156 156 156 294 294 294 F F F 4 080-099 156 156 156 294 294 294 F F F 5 100-119 153 153 153 288 288 288 F F F 6 120-139 153 153 153 288 288 288 F F F 7 140-159 156 156 156 294 294 294 F F F 8 160-179 156 156 156 294 294 294 F F F 9 180-199 156 156 156 294 294 294 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 155 155 155 291 291 291 F F F 14 280-299 156 158 158 292 299 299 F T T <--- 15 300-319 154 158 158 290 299 299 F F F <--- 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 156 158 158 294 299 299 F F F <--- 19 380-399 152 152 152 285 285 285 F F F 20 400-419 152 158 158 285 296 296 F F F <--- 21 420-439 153 158 158 288 298 298 F F F <--- 22 440-459 156 156 156 294 294 294 F T T <--- 23 460-479 152 152 152 287 287 287 F T T <--- 24 480-499 155 158 158 291 298 298 F F F <--- 25 500-519 156 158 158 294 296 296 T F F <--- 26 520-539 143 143 143 270 270 270 F F F 27 540-559 156 158 158 294 298 298 F F F <--- 28 560-579 148 158 158 279 299 299 F F F <--- 29 580-599 153 153 153 288 288 288 F F F Max gen: 158, Max flush: 299 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 294 294 294 F F F 1 020-039 159 159 159 300 300 300 F F F 2 040-059 155 155 155 291 291 291 F F F 3 060-079 156 156 156 294 294 294 F F F 4 080-099 156 156 156 294 294 294 F F F 5 100-119 153 153 153 288 288 288 F F F 6 120-139 153 153 153 288 288 288 F F F 7 140-159 156 156 156 294 294 294 F F F 8 160-179 156 156 156 294 294 294 F F F 9 180-199 156 156 156 294 294 294 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 155 155 155 291 291 291 F F F 14 280-299 159 159 159 300 300 300 F F F 15 300-319 158 158 158 299 299 299 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 158 158 158 299 299 299 F F F 19 380-399 152 152 152 285 285 285 F F F 20 400-419 158 158 158 296 296 296 F F F 21 420-439 158 158 158 298 298 298 F F F 22 440-459 159 159 159 300 300 300 F F F 23 460-479 159 159 159 300 300 300 F F F 24 480-499 158 158 158 298 298 298 F F F 25 500-519 158 158 158 296 296 296 F F F 26 520-539 143 143 143 270 270 270 F F F 27 540-559 158 158 158 298 298 298 F F F 28 560-579 158 158 158 299 299 299 F F F 29 580-599 153 153 153 288 288 288 F F F Max gen: 159, Max flush: 300 Begin loop 80 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.776803936Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.785111082Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.785404689Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.785435445Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.785443873Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.785455402Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777} {"msg":"Crucible f9e4d7b1-a1dc-4667-b255-e0be755aaa1b has session id: ebbcbedb-beec-4de4-98ed-403097a630e0","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.785723922Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Upstairs opts: Upstairs UUID: f9e4d7b1-a1dc-4667-b255-e0be755aaa1b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.785781524Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Crucible stats registered with UUID: f9e4d7b1-a1dc-4667-b255-e0be755aaa1b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.785791782Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:05.785800841Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.785970496Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"io task","client":"1","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.78601217Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"io task","client":"0","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786023499Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"io task","client":"2","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786120814Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"task":"crutest"} {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786134762Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777} {"msg":"f9e4d7b1-a1dc-4667-b255-e0be755aaa1b active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786175576Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786185105Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"client":"0","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786194764Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"client":"1","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786204412Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"client":"2","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786213721Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"io task","client":"2","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786223769Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"io task","client":"0","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786233558Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"io task","client":"1","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:52356)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786247266Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"io task","client":"0","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:62753)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786299178Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"io task","client":"2","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"ds_connection connected from Ok(127.0.0.1:35954)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786450726Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"io task","client":"1","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786932736Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"client":"0","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786958932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"client":"1","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.786975939Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"client":"2","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.78724339Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787253219Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"[0]R flush_numbers[0..12]: [294, 300, 291, 294, 294, 288, 288, 294, 294, 294, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787290123Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"[0]R generation[0..12]: [156, 159, 155, 156, 156, 153, 153, 156, 156, 156, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787301532Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.78731238Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"[1]R flush_numbers[0..12]: [294, 300, 291, 294, 294, 288, 288, 294, 294, 294, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787323049Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"[1]R generation[0..12]: [156, 159, 155, 156, 156, 153, 153, 156, 156, 156, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787333387Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787344356Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"[2]R flush_numbers[0..12]: [294, 300, 291, 294, 294, 288, 288, 294, 294, 294, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787354654Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"[2]R generation[0..12]: [156, 159, 155, 156, 156, 153, 153, 156, 156, 156, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787365073Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787375901Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.78738533Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787396498Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Next flush: 301","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787405807Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787415215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"":"downstairs","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787424894Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787433732Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"client":"0","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787443451Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"client":"1","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.78745302Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"client":"2","":"downstairs","session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787462818Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787475186Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"f9e4d7b1-a1dc-4667-b255-e0be755aaa1b is now active with session: ebbcbedb-beec-4de4-98ed-403097a630e0","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787485135Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787494334Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787501853Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777,"session_id":"ebbcbedb-beec-4de4-98ed-403097a630e0"} {"msg":"Activated sub_volume f9e4d7b1-a1dc-4667-b255-e0be755aaa1b","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:05.787510841Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 504 len 8192 data: 5 10 02/30 Write block 137 len 24576 data: 9 11 16 20 18 16 03/30 Read block 504 len 12288 04/30 Write block 66 len 28672 data: 13 13 12 13 12 14 15 05/30 Read block 81 len 12288 06/30 Read block 103 len 28672 07/30 Write block 276 len 40960 data: 8 8 6 13 12 13 10 13 13 14 08/30 Read block 165 len 4096 09/30 Read block 282 len 40960 10/30 Read block 90 len 20480 11/30 Read block 555 len 20480 12/30 Flush 13/30 Write block 134 len 12288 data: 12 10 8 14/30 Write block 580 len 36864 data: 11 10 10 11 12 12 10 13 12 15/30 Read block 118 len 32768 16/30 Read block 391 len 32768 17/30 Read block 389 len 16384 18/30 Read block 443 len 32768 19/30 Read block 377 len 32768 20/30 Read block 51 len 8192 21/30 Read block 25 len 28672 22/30 Write block 513 len 40960 data: 12 14 12 11 12 10 11 10 12 12 23/30 Read block 26 len 40960 24/30 Flush 25/30 Flush 26/30 Write block 491 len 12288 data: 9 7 7 27/30 Read block 109 len 4096 28/30 Read block 105 len 36864 29/30 Write block 362 len 8192 data: 4 5 30/30 Write block 131 len 8192 data: 14 14 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Read 3 Done Done Sent false 1003 Acked Write 7 Done Done Sent false 1004 Acked Read 3 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Write 10 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 3 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Read 8 Done Done Sent false 1016 Acked Read 4 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Read 8 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 28 32 Done 28 28 2 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:05.796588564Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1777} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 294 294 294 F F F 1 020-039 159 159 159 300 300 300 F F F 2 040-059 155 155 155 291 291 291 F F F 3 060-079 160 160 156 301 301 294 F F F <--- 4 080-099 156 156 156 294 294 294 F F F 5 100-119 153 153 153 288 288 288 F F F 6 120-139 160 160 153 302 302 288 T T T <--- 7 140-159 160 160 156 301 301 294 F F T <--- 8 160-179 156 156 156 294 294 294 F F F 9 180-199 156 156 156 294 294 294 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 160 160 155 301 301 291 F F F <--- 14 280-299 160 160 159 301 301 300 F F F <--- 15 300-319 158 158 158 299 299 299 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 158 158 158 299 299 299 T T F <--- 19 380-399 152 152 152 285 285 285 F F F 20 400-419 158 158 158 296 296 296 F F F 21 420-439 158 158 158 298 298 298 F F F 22 440-459 159 159 159 300 300 300 F F F 23 460-479 159 159 159 300 300 300 F F F 24 480-499 158 158 158 298 298 298 T T F <--- 25 500-519 160 160 158 302 302 296 F F T <--- 26 520-539 160 160 143 302 302 270 F F F <--- 27 540-559 158 158 158 298 298 298 F F F 28 560-579 158 158 158 299 299 299 F F F 29 580-599 160 160 153 302 302 288 F F F <--- Max gen: 160, Max flush: 302 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 156 156 156 294 294 294 F F F 1 020-039 159 159 159 300 300 300 F F F 2 040-059 155 155 155 291 291 291 F F F 3 060-079 160 160 160 301 301 301 F F F 4 080-099 156 156 156 294 294 294 F F F 5 100-119 153 153 153 288 288 288 F F F 6 120-139 161 161 161 303 303 303 F F F 7 140-159 160 160 160 301 301 301 F F F 8 160-179 156 156 156 294 294 294 F F F 9 180-199 156 156 156 294 294 294 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 160 160 160 301 301 301 F F F 14 280-299 160 160 160 301 301 301 F F F 15 300-319 158 158 158 299 299 299 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 F F F 18 360-379 161 161 161 303 303 303 F F F 19 380-399 152 152 152 285 285 285 F F F 20 400-419 158 158 158 296 296 296 F F F 21 420-439 158 158 158 298 298 298 F F F 22 440-459 159 159 159 300 300 300 F F F 23 460-479 159 159 159 300 300 300 F F F 24 480-499 161 161 161 303 303 303 F F F 25 500-519 160 160 160 302 302 302 F F F 26 520-539 160 160 160 302 302 302 F F F 27 540-559 158 158 158 298 298 298 F F F 28 560-579 158 158 158 299 299 299 F F F 29 580-599 160 160 160 302 302 302 F F F Max gen: 161, Max flush: 303 Begin loop 81 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.042013664Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.050168942Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.050448861Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.050476977Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.050487826Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.050499244Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785} {"msg":"Crucible aca453f1-cd0d-4f82-87bb-b3fc17fec01a has session id: b4c1dd3a-25d7-4a22-b3cd-48f3f972a662","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.05086735Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Upstairs opts: Upstairs UUID: aca453f1-cd0d-4f82-87bb-b3fc17fec01a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.050897136Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Crucible stats registered with UUID: aca453f1-cd0d-4f82-87bb-b3fc17fec01a","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.050908364Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:10.050917453Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051058762Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"io task","client":"2","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051082069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"io task","client":"1","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051092067Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"io task","client":"0","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051232297Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"task":"crutest"} {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051249634Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785} {"msg":"aca453f1-cd0d-4f82-87bb-b3fc17fec01a active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051273331Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051282679Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"client":"0","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051292318Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"client":"1","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051301907Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"client":"2","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051313425Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"io task","client":"2","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051323253Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"io task","client":"0","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051332972Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"io task","client":"1","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"ds_connection connected from Ok(127.0.0.1:62579)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051388144Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"io task","client":"1","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"ds_connection connected from Ok(127.0.0.1:60059)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051398852Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"io task","client":"2","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"ds_connection connected from Ok(127.0.0.1:56829)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.051410781Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"io task","client":"0","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052037399Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"client":"1","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052067895Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"client":"0","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052116697Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"client":"2","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052330626Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052347844Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"[0]R flush_numbers[0..12]: [294, 300, 291, 301, 294, 288, 303, 301, 294, 294, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052402646Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"[0]R generation[0..12]: [156, 159, 155, 160, 156, 153, 161, 160, 156, 156, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052414014Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052424972Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"[1]R flush_numbers[0..12]: [294, 300, 291, 301, 294, 288, 303, 301, 294, 294, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052437791Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"[1]R generation[0..12]: [156, 159, 155, 160, 156, 153, 161, 160, 156, 156, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052448319Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052459097Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"[2]R flush_numbers[0..12]: [294, 300, 291, 301, 294, 288, 303, 301, 294, 294, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052469466Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"[2]R generation[0..12]: [156, 159, 155, 160, 156, 153, 161, 160, 156, 156, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052479894Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052490733Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Max found gen is 162","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052500111Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.05250972Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Next flush: 304","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052519359Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052528727Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"":"downstairs","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052538616Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052547404Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"client":"0","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052557323Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"client":"1","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052567222Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"client":"2","":"downstairs","session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.05257714Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052587889Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"Wait for a query_work_queue command to finish before sending IO session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"aca453f1-cd0d-4f82-87bb-b3fc17fec01a is now active with session: b4c1dd3a-25d7-4a22-b3cd-48f3f972a662","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052601577Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052610245Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785,"session_id":"b4c1dd3a-25d7-4a22-b3cd-48f3f972a662"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052619204Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785} {"msg":"Activated sub_volume aca453f1-cd0d-4f82-87bb-b3fc17fec01a","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:10.052627053Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 15 len 12288 data: 14 13 14 03/30 Read block 63 len 36864 04/30 Write block 501 len 40960 data: 9 8 6 6 11 9 8 12 15 16 05/30 Read block 538 len 20480 06/30 Write block 422 len 28672 data: 14 14 12 13 14 15 12 07/30 Write block 372 len 24576 data: 11 12 14 13 15 16 08/30 Read block 441 len 12288 09/30 Flush 10/30 Write block 180 len 28672 data: 20 19 19 19 15 12 13 11/30 Write block 446 len 28672 data: 16 14 13 14 11 9 9 12/30 Read block 172 len 28672 13/30 Read block 571 len 24576 14/30 Read block 380 len 20480 15/30 Write block 1 len 28672 data: 5 8 8 10 10 10 10 16/30 Read block 403 len 28672 17/30 Write block 285 len 8192 data: 15 13 18/30 Read block 585 len 4096 19/30 Flush 20/30 Flush 21/30 Write block 455 len 28672 data: 15 16 16 19 18 17 15 22/30 Write block 58 len 28672 data: 13 12 10 12 12 10 13 23/30 Write block 291 len 8192 data: 15 14 24/30 Read block 293 len 4096 25/30 Write block 533 len 4096 data: 16 26/30 Read block 513 len 16384 27/30 Write block 435 len 40960 data: 23 22 20 21 16 14 15 17 15 15 28/30 Write block 533 len 40960 data: 17 15 16 14 15 14 14 12 13 13 29/30 Write block 542 len 32768 data: 14 11 9 8 9 7 9 10 30/30 Write block 358 len 12288 data: 12 8 10 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Done Done Done false 1002 Acked Read 9 Done Done Sent false 1003 Acked Write 10 Done Done Sent false 1004 Acked Read 5 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Read 3 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 7 Done Done Sent false 1010 Acked Write 7 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Write 10 Done Sent Sent false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 4 28 35 Done 26 25 1 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:10.061167884Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1785} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 162 162 156 306 306 294 F F T <--- 1 020-039 159 159 159 300 300 300 F F F 2 040-059 155 155 155 291 291 291 T T F <--- 3 060-079 160 160 160 301 301 301 T T F <--- 4 080-099 156 156 156 294 294 294 F F F 5 100-119 153 153 153 288 288 288 F F F 6 120-139 161 161 161 303 303 303 F F F 7 140-159 160 160 160 301 301 301 F F F 8 160-179 156 156 156 294 294 294 F F F 9 180-199 162 162 156 306 306 294 F F F <--- 10 200-219 139 139 139 258 258 258 F F F 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 160 160 160 301 301 301 F F F 14 280-299 162 162 160 306 306 301 T T F <--- 15 300-319 158 158 158 299 299 299 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 141 141 141 265 265 265 T T F <--- 18 360-379 162 162 161 305 305 303 T T F <--- 19 380-399 152 152 152 285 285 285 F F F 20 400-419 158 158 158 296 296 296 F F F 21 420-439 162 162 158 305 305 298 T T F <--- 22 440-459 162 162 159 306 306 300 T T F <--- 23 460-479 159 159 159 300 300 300 T T F <--- 24 480-499 161 161 161 303 303 303 F F F 25 500-519 162 162 160 305 305 302 F F F <--- 26 520-539 160 160 160 302 302 302 T T F <--- 27 540-559 158 158 158 298 298 298 T T F <--- 28 560-579 158 158 158 299 299 299 F F F 29 580-599 160 160 160 302 302 302 F F F Max gen: 162, Max flush: 306 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 162 162 162 306 306 306 F F F 1 020-039 159 159 159 300 300 300 F F F 2 040-059 163 163 163 307 307 307 F F F 3 060-079 163 163 163 307 307 307 F F F 4 080-099 156 156 156 294 294 294 F F F 5 100-119 153 153 153 288 288 288 F F F 6 120-139 161 161 161 303 303 303 F F F 7 140-159 160 160 160 301 301 301 F F F 8 160-179 156 156 156 294 294 294 F F F 9 180-199 162 162 162 306 306 306 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 160 160 160 301 301 301 F F F 14 280-299 163 163 163 307 307 307 F F F 15 300-319 158 158 158 299 299 299 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 163 163 163 307 307 307 F F F 18 360-379 163 163 163 307 307 307 F F F 19 380-399 152 152 152 285 285 285 F F F 20 400-419 158 158 158 296 296 296 F F F 21 420-439 163 163 163 307 307 307 F F F 22 440-459 163 163 163 307 307 307 F F F 23 460-479 163 163 163 307 307 307 F F F 24 480-499 161 161 161 303 303 303 F F F 25 500-519 162 162 162 305 305 305 F F F 26 520-539 163 163 163 307 307 307 F F F 27 540-559 163 163 163 307 307 307 F F F 28 560-579 158 158 158 299 299 299 F F F 29 580-599 160 160 160 302 302 302 F F F Max gen: 163, Max flush: 307 Begin loop 82 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.317205484Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.325396017Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.325691673Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.325719899Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.325728288Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.325739736Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793} {"msg":"Crucible 2868d8d5-1e5a-470d-9129-d4bc8cebe6ff has session id: 22bb7eb1-4154-46e2-85de-28bfa6be9f64","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326102763Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Upstairs opts: Upstairs UUID: 2868d8d5-1e5a-470d-9129-d4bc8cebe6ff, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326132079Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Crucible stats registered with UUID: 2868d8d5-1e5a-470d-9129-d4bc8cebe6ff","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326143187Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:14.326154066Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326307863Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"io task","client":"1","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.32633039Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"io task","client":"2","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326340579Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"io task","client":"0","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.32646459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"task":"crutest"} {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326488567Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793} {"msg":"2868d8d5-1e5a-470d-9129-d4bc8cebe6ff active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326501315Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326509994Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"client":"0","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326519572Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"client":"1","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326529041Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"client":"2","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.32653832Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"io task","client":"2","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326568845Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"io task","client":"0","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326590452Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"io task","client":"1","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"ds_connection connected from Ok(127.0.0.1:34076)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.32660136Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"io task","client":"0","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"ds_connection connected from Ok(127.0.0.1:48943)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326611949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"io task","client":"1","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"ds_connection connected from Ok(127.0.0.1:36225)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.326627317Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"io task","client":"2","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327349601Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"client":"1","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327379927Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"client":"0","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.32749087Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"client":"2","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327685922Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327703489Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"[0]R flush_numbers[0..12]: [306, 300, 307, 307, 294, 288, 303, 301, 294, 306, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327756552Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"[0]R generation[0..12]: [162, 159, 163, 163, 156, 153, 161, 160, 156, 162, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.32776777Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327778658Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"[1]R flush_numbers[0..12]: [306, 300, 307, 307, 294, 288, 303, 301, 294, 306, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327789187Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"[1]R generation[0..12]: [162, 159, 163, 163, 156, 153, 161, 160, 156, 162, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327799585Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327810364Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"[2]R flush_numbers[0..12]: [306, 300, 307, 307, 294, 288, 303, 301, 294, 306, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327820872Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"[2]R generation[0..12]: [162, 159, 163, 163, 156, 153, 161, 160, 156, 162, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327831331Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327842089Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327853207Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327862716Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Next flush: 308","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327871975Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327881173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"":"downstairs","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327890842Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327899561Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"client":"0","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327909169Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"client":"1","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327918738Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"client":"2","":"downstairs","session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327928546Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:16:14.327937565Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"2868d8d5-1e5a-470d-9129-d4bc8cebe6ff is now active with session: 22bb7eb1-4154-46e2-85de-28bfa6be9f64","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327950423Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327959412Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327966981Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793,"session_id":"22bb7eb1-4154-46e2-85de-28bfa6be9f64"} {"msg":"Activated sub_volume 2868d8d5-1e5a-470d-9129-d4bc8cebe6ff","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:14.327975959Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 261 len 40960 data: 10 13 15 16 17 19 19 17 18 15 02/30 Read block 449 len 4096 03/30 Read block 22 len 16384 04/30 Read block 74 len 36864 05/30 Write block 583 len 4096 data: 12 06/30 Read block 311 len 4096 07/30 Read block 182 len 32768 08/30 Read block 132 len 28672 09/30 Write block 576 len 8192 data: 16 14 10/30 Write block 547 len 24576 data: 8 10 11 10 11 10 11/30 Write block 532 len 16384 data: 14 18 16 17 12/30 Write block 39 len 16384 data: 17 19 20 19 13/30 Read block 403 len 20480 14/30 Read block 326 len 24576 15/30 Write block 271 len 16384 data: 16 15 15 12 16/30 Write block 79 len 32768 data: 11 10 16 18 15 14 16 14 17/30 Read block 28 len 8192 18/30 Read block 563 len 24576 19/30 Write block 569 len 4096 data: 11 20/30 Read block 420 len 20480 21/30 Flush 22/30 Write block 95 len 28672 data: 12 14 17 17 16 16 16 23/30 Write block 305 len 24576 data: 10 13 10 11 14 14 24/30 Read block 187 len 16384 25/30 Read block 173 len 24576 26/30 Read block 192 len 20480 27/30 Read block 486 len 24576 28/30 Write block 351 len 28672 data: 12 14 14 14 15 15 14 29/30 Write block 427 len 12288 data: 16 13 12 30/30 Read block 197 len 8192 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Write 1 Done Done Done false 1005 Acked Read 1 Done Done Done false 1006 Acked Read 8 Done Done Done false 1007 Acked Read 7 Done Done Done false 1008 Acked Write 2 Done Done Done false 1009 Acked Write 6 Done Done Done false 1010 Acked Write 4 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Read 6 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Write 1 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Read 5 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Write 7 Done Done Sent false 1028 Acked Write 3 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 20 20 Done 30 30 10 70 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:14.33659118Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1793} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 162 162 162 306 306 306 F F F 1 020-039 164 164 159 308 308 300 F F F <--- 2 040-059 164 164 163 308 308 307 F F F <--- 3 060-079 164 164 163 308 308 307 F F F <--- 4 080-099 164 164 156 308 308 294 T T F <--- 5 100-119 153 153 153 288 288 288 T T F <--- 6 120-139 161 161 161 303 303 303 F F F 7 140-159 160 160 160 301 301 301 F F F 8 160-179 156 156 156 294 294 294 F F F 9 180-199 162 162 162 306 306 306 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 164 164 160 308 308 301 F F T <--- 14 280-299 163 163 163 307 307 307 F F F 15 300-319 158 158 158 299 299 299 T T F <--- 16 320-339 150 150 150 280 280 280 F F F 17 340-359 163 163 163 307 307 307 T T F <--- 18 360-379 163 163 163 307 307 307 F F F 19 380-399 152 152 152 285 285 285 F F F 20 400-419 158 158 158 296 296 296 F F F 21 420-439 163 163 163 307 307 307 T T F <--- 22 440-459 163 163 163 307 307 307 F F F 23 460-479 163 163 163 307 307 307 F F F 24 480-499 161 161 161 303 303 303 F F F 25 500-519 162 162 162 305 305 305 F F F 26 520-539 164 164 163 308 308 307 F F F <--- 27 540-559 164 164 163 308 308 307 F F T <--- 28 560-579 164 164 158 308 308 299 F F T <--- 29 580-599 164 164 160 308 308 302 F F T <--- Max gen: 164, Max flush: 308 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 162 162 162 306 306 306 F F F 1 020-039 164 164 164 308 308 308 F F F 2 040-059 164 164 164 308 308 308 F F F 3 060-079 164 164 164 308 308 308 F F F 4 080-099 165 165 165 309 309 309 F F F 5 100-119 165 165 165 309 309 309 F F F 6 120-139 161 161 161 303 303 303 F F F 7 140-159 160 160 160 301 301 301 F F F 8 160-179 156 156 156 294 294 294 F F F 9 180-199 162 162 162 306 306 306 F F F 10 200-219 139 139 139 258 258 258 F F F 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 164 164 164 308 308 308 F F F 14 280-299 163 163 163 307 307 307 F F F 15 300-319 165 165 165 309 309 309 F F F 16 320-339 150 150 150 280 280 280 F F F 17 340-359 165 165 165 309 309 309 F F F 18 360-379 163 163 163 307 307 307 F F F 19 380-399 152 152 152 285 285 285 F F F 20 400-419 158 158 158 296 296 296 F F F 21 420-439 165 165 165 309 309 309 F F F 22 440-459 163 163 163 307 307 307 F F F 23 460-479 163 163 163 307 307 307 F F F 24 480-499 161 161 161 303 303 303 F F F 25 500-519 162 162 162 305 305 305 F F F 26 520-539 164 164 164 308 308 308 F F F 27 540-559 164 164 164 308 308 308 F F F 28 560-579 164 164 164 308 308 308 F F F 29 580-599 164 164 164 308 308 308 F F F Max gen: 165, Max flush: 309 Begin loop 83 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.591287255Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:18.598540155Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.598592928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.605584146Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.605848627Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.605883382Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.605892121Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.605903519Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801} {"msg":"Crucible 2de437e3-74ad-4cce-b5e4-704f6e3be36f has session id: 461b941d-9718-460b-902e-180ca5e4f141","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606245459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Upstairs opts: Upstairs UUID: 2de437e3-74ad-4cce-b5e4-704f6e3be36f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606277294Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Crucible stats registered with UUID: 2de437e3-74ad-4cce-b5e4-704f6e3be36f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606288503Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:18.606297761Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606435271Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"io task","client":"2","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606458128Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"io task","client":"0","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606468287Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"io task","client":"1","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.60657947Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"task":"crutest"} {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606604157Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801} {"msg":"2de437e3-74ad-4cce-b5e4-704f6e3be36f active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606623254Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606632103Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"client":"0","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606641981Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"client":"1","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.60665148Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"client":"2","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606661008Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"io task","client":"2","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606686035Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"io task","client":"1","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"ds_connection connected from Ok(127.0.0.1:34105)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606741207Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"io task","client":"2","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606767633Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"io task","client":"0","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"ds_connection connected from Ok(127.0.0.1:33598)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606778961Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"io task","client":"1","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"ds_connection connected from Ok(127.0.0.1:45839)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.606804087Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"io task","client":"0","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607431836Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"client":"2","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607452843Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"client":"0","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607483348Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"client":"1","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607740921Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607774256Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"[0]R flush_numbers[0..12]: [306, 308, 308, 308, 309, 309, 303, 301, 294, 306, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607837586Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"[0]R generation[0..12]: [162, 164, 164, 164, 165, 165, 161, 160, 156, 162, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607851614Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607862753Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"[1]R flush_numbers[0..12]: [306, 308, 308, 308, 309, 309, 303, 301, 294, 306, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607873341Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"[1]R generation[0..12]: [162, 164, 164, 164, 165, 165, 161, 160, 156, 162, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.60788396Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607894998Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"[2]R flush_numbers[0..12]: [306, 308, 308, 308, 309, 309, 303, 301, 294, 306, 258, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607905447Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"[2]R generation[0..12]: [162, 164, 164, 164, 165, 165, 161, 160, 156, 162, 139, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607915755Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607926503Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Max found gen is 166","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607935772Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Generation requested: 166 >= found:166","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607945421Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Next flush: 310","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607954859Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607964028Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"":"downstairs","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607973607Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607982375Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"client":"0","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.607992974Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"client":"1","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.608002402Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"client":"2","":"downstairs","session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.608012211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.60802125Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"2de437e3-74ad-4cce-b5e4-704f6e3be36f is now active with session: 461b941d-9718-460b-902e-180ca5e4f141","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:16:18.608031018Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.608043716Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.608051255Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801,"session_id":"461b941d-9718-460b-902e-180ca5e4f141"} {"msg":"Activated sub_volume 2de437e3-74ad-4cce-b5e4-704f6e3be36f","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:18.608060404Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 528 len 16384 data: 13 12 11 12 03/30 Read block 330 len 16384 04/30 Write block 329 len 12288 data: 11 14 15 05/30 Read block 131 len 36864 06/30 Write block 387 len 12288 data: 9 12 12 07/30 Flush 08/30 Write block 573 len 36864 data: 11 12 14 17 15 14 13 12 11 09/30 Read block 349 len 16384 10/30 Write block 158 len 24576 data: 10 12 11 10 11 19 11/30 Read block 407 len 24576 12/30 Read block 581 len 20480 13/30 Read block 407 len 16384 14/30 Write block 201 len 8192 data: 10 9 15/30 Read block 490 len 20480 16/30 Write block 343 len 28672 data: 13 12 15 14 13 9 11 17/30 Flush 18/30 Write block 173 len 8192 data: 12 12 19/30 Write block 540 len 40960 data: 13 14 15 12 10 9 10 9 11 12 20/30 Read block 248 len 28672 21/30 Write block 393 len 32768 data: 16 11 11 11 10 12 11 11 22/30 Flush 23/30 Write block 557 len 36864 data: 16 15 14 14 14 12 11 12 11 24/30 Write block 32 len 28672 data: 9 9 8 9 10 16 17 25/30 Read block 465 len 40960 26/30 Flush 27/30 Write block 198 len 36864 data: 15 13 11 11 10 9 13 11 10 28/30 Write block 194 len 32768 data: 17 17 15 15 16 14 12 12 29/30 Write block 484 len 8192 data: 10 10 30/30 Write block 55 len 8192 data: 14 17 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:23 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1007 Acked Write 9 Done Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Write 2 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 9 Sent Sent Sent false 1027 Acked Write 8 Sent Sent Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 22 4 4 30 Done 1 19 19 39 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1006)) Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: 1006 1005 1004 1003 1002 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:18.617562215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1801} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 162 162 162 306 306 306 F F F 1 020-039 164 166 166 308 314 314 F F F <--- 2 040-059 164 164 164 308 308 308 F T T <--- 3 060-079 164 164 164 308 308 308 F F F 4 080-099 165 165 165 309 309 309 F F F 5 100-119 165 165 165 309 309 309 F F F 6 120-139 161 161 161 303 303 303 F F F 7 140-159 160 166 166 301 312 312 F F F <--- 8 160-179 156 166 166 294 313 313 F F F <--- 9 180-199 162 162 162 306 306 306 F T T <--- 10 200-219 139 166 166 258 312 312 F T T <--- 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 164 164 164 308 308 308 F F F 14 280-299 163 163 163 307 307 307 F F F 15 300-319 165 165 165 309 309 309 F F F 16 320-339 166 166 166 311 311 311 F F F 17 340-359 165 166 166 309 312 312 F F F <--- 18 360-379 163 163 163 307 307 307 F F F 19 380-399 166 166 166 311 313 313 F F F <--- 20 400-419 158 166 166 296 313 313 F F F <--- 21 420-439 165 165 165 309 309 309 F F F 22 440-459 163 163 163 307 307 307 F F F 23 460-479 163 163 163 307 307 307 F F F 24 480-499 161 161 161 303 303 303 F T T <--- 25 500-519 162 162 162 305 305 305 F F F 26 520-539 166 166 166 311 311 311 F F F 27 540-559 164 166 166 308 314 314 F F F <--- 28 560-579 164 166 166 308 314 314 T F F <--- 29 580-599 164 166 166 308 312 312 T F F <--- Max gen: 166, Max flush: 314 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 162 162 162 306 306 306 F F F 1 020-039 166 166 166 314 314 314 F F F 2 040-059 167 167 167 315 315 315 F F F 3 060-079 164 164 164 308 308 308 F F F 4 080-099 165 165 165 309 309 309 F F F 5 100-119 165 165 165 309 309 309 F F F 6 120-139 161 161 161 303 303 303 F F F 7 140-159 166 166 166 312 312 312 F F F 8 160-179 166 166 166 313 313 313 F F F 9 180-199 167 167 167 315 315 315 F F F 10 200-219 167 167 167 315 315 315 F F F 11 220-239 157 157 157 295 295 295 F F F 12 240-259 155 155 155 291 291 291 F F F 13 260-279 164 164 164 308 308 308 F F F 14 280-299 163 163 163 307 307 307 F F F 15 300-319 165 165 165 309 309 309 F F F 16 320-339 166 166 166 311 311 311 F F F 17 340-359 166 166 166 312 312 312 F F F 18 360-379 163 163 163 307 307 307 F F F 19 380-399 166 166 166 313 313 313 F F F 20 400-419 166 166 166 313 313 313 F F F 21 420-439 165 165 165 309 309 309 F F F 22 440-459 163 163 163 307 307 307 F F F 23 460-479 163 163 163 307 307 307 F F F 24 480-499 167 167 167 315 315 315 F F F 25 500-519 162 162 162 305 305 305 F F F 26 520-539 166 166 166 311 311 311 F F F 27 540-559 166 166 166 314 314 314 F F F 28 560-579 166 166 166 314 314 314 F F F 29 580-599 166 166 166 312 312 312 F F F Max gen: 167, Max flush: 315 Begin loop 84 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.90673694Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.915000443Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.9152898Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.915317326Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.915326195Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.915337643Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809} {"msg":"Crucible 2b69d939-1655-4fb0-bb77-9bc3f88d4527 has session id: 40d25a96-ffec-4ecb-8de6-9a34a0162748","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.915686282Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Upstairs opts: Upstairs UUID: 2b69d939-1655-4fb0-bb77-9bc3f88d4527, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.915715288Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Crucible stats registered with UUID: 2b69d939-1655-4fb0-bb77-9bc3f88d4527","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.915728136Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:22.915737295Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.915889663Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"io task","client":"1","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.915924518Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"io task","client":"0","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.915935386Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"io task","client":"2","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916061018Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"task":"crutest"} {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916085914Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809} {"msg":"2b69d939-1655-4fb0-bb77-9bc3f88d4527 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.91611127Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916120939Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"client":"0","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916130707Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"client":"1","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916140326Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"client":"2","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916149875Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"io task","client":"2","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916159873Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"io task","client":"0","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916203847Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"io task","client":"1","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"ds_connection connected from Ok(127.0.0.1:49372)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916222864Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"io task","client":"2","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"ds_connection connected from Ok(127.0.0.1:35941)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916233422Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"io task","client":"0","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"ds_connection connected from Ok(127.0.0.1:38890)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916275366Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"io task","client":"1","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916820547Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"client":"0","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.916838654Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"client":"2","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917119163Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917196002Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917229387Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"client":"1","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"[0]R flush_numbers[0..12]: [306, 314, 315, 308, 309, 309, 303, 312, 313, 315, 315, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917503707Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"[0]R generation[0..12]: [162, 166, 167, 164, 165, 165, 161, 166, 166, 167, 167, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917514745Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917525604Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"[1]R flush_numbers[0..12]: [306, 314, 315, 308, 309, 309, 303, 312, 313, 315, 315, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917536252Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"[1]R generation[0..12]: [162, 166, 167, 164, 165, 165, 161, 166, 166, 167, 167, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.9175465Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917557529Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"[2]R flush_numbers[0..12]: [306, 314, 315, 308, 309, 309, 303, 312, 313, 315, 315, 295]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917568027Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"[2]R generation[0..12]: [162, 166, 167, 164, 165, 165, 161, 166, 166, 167, 167, 157]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917580126Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917590924Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Max found gen is 168","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917600123Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917609771Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Next flush: 316","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.91761894Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917628159Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"":"downstairs","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917637567Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917646296Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"client":"0","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917655814Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"client":"1","":"downstairs","session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917665303Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"client":"2","":"downstairs","session_idWait for a query_work_queue command to finish before sending IO ":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917678991Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.91768796Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"2b69d939-1655-4fb0-bb77-9bc3f88d4527 is now active with session: 40d25a96-ffec-4ecb-8de6-9a34a0162748","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917697798Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917706837Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917714366Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809,"session_id":"40d25a96-ffec-4ecb-8de6-9a34a0162748"} {"msg":"Activated sub_volume 2b69d939-1655-4fb0-bb77-9bc3f88d4527","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:22.917723445Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 531 len 36864 data: 13 15 19 17 18 15 16 15 15 03/30 Write block 37 len 36864 data: 17 18 18 20 21 20 19 16 13 04/30 Read block 216 len 12288 05/30 Write block 130 len 40960 data: 14 15 15 13 13 11 9 10 12 17 06/30 Write block 199 len 32768 data: 15 13 13 11 10 14 12 11 07/30 Read block 47 len 8192 08/30 Write block 158 len 12288 data: 11 13 12 09/30 Read block 255 len 20480 10/30 Write block 185 len 36864 data: 13 14 11 10 9 12 13 15 16 11/30 Write block 216 len 40960 data: 7 7 8 8 6 9 12 16 14 14 12/30 Write block 77 len 32768 data: 10 12 12 11 17 19 16 15 13/30 Write block 248 len 20480 data: 12 12 13 11 9 14/30 Read block 319 len 20480 15/30 Read block 216 len 40960 16/30 Read block 46 len 24576 17/30 Read block 525 len 8192 18/30 Write block 228 len 16384 data: 16 17 17 14 19/30 Read block 173 len 32768 20/30 Write block 410 len 40960 data: 10 9 8 10 9 9 9 10 11 9 21/30 Read block 176 len 36864 22/30 Read block 311 len 20480 23/30 Write block 436 len 40960 data: 23 21 22 17 15 16 18 16 16 17 24/30 Read block 69 len 32768 25/30 Read block 0 len 16384 26/30 Read block 59 len 32768 27/30 Write block 578 len 32768 data: 15 14 13 12 11 13 13 13 28/30 Write block 555 len 40960 data: 14 15 17 16 15 15 15 13 12 13 29/30 Read block 523 len 16384 30/30 Read block 533 len 32768 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Done Sent Done false 1001 Acked Write 9 Done Sent Done false 1002 Acked Write 9 Done Sent Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Write 10 Done Sent Done false 1005 Acked Write 8 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Write 3 Done Sent Done false 1008 Acked Read 5 Done Sent Done false 1009 Acked Write 9 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Write 5 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Write 10 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Write 10 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 1 31 Done 30 0 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) None Some(JobId(1000)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:22.926957955Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1809} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 162 162 162 306 306 306 F F F 1 020-039 166 166 166 314 314 314 T F T <--- 2 040-059 167 167 167 315 315 315 T F T <--- 3 060-079 164 164 164 308 308 308 T F T <--- 4 080-099 165 165 165 309 309 309 T F T <--- 5 100-119 165 165 165 309 309 309 F F F 6 120-139 161 161 161 303 303 303 T F T <--- 7 140-159 166 166 166 312 312 312 T F T <--- 8 160-179 166 166 166 313 313 313 T F T <--- 9 180-199 167 167 167 315 315 315 T F T <--- 10 200-219 167 167 167 315 315 315 T F T <--- 11 220-239 157 157 157 295 295 295 T F T <--- 12 240-259 155 155 155 291 291 291 T F T <--- 13 260-279 164 164 164 308 308 308 F F F 14 280-299 163 163 163 307 307 307 F F F 15 300-319 165 165 165 309 309 309 F F F 16 320-339 166 166 166 311 311 311 F F F 17 340-359 166 166 166 312 312 312 F F F 18 360-379 163 163 163 307 307 307 F F F 19 380-399 166 166 166 313 313 313 F F F 20 400-419 166 166 166 313 313 313 T F T <--- 21 420-439 165 165 165 309 309 309 T F T <--- 22 440-459 163 163 163 307 307 307 T F T <--- 23 460-479 163 163 163 307 307 307 F F F 24 480-499 167 167 167 315 315 315 F F F 25 500-519 162 162 162 305 305 305 F F F 26 520-539 166 166 166 311 311 311 T F T <--- 27 540-559 166 166 166 314 314 314 T F T <--- 28 560-579 166 166 166 314 314 314 T F T <--- 29 580-599 166 166 166 312 312 312 T F T <--- Max gen: 167, Max flush: 315 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 162 162 162 306 306 306 F F F 1 020-039 168 168 168 316 316 316 F F F 2 040-059 168 168 168 316 316 316 F F F 3 060-079 168 168 168 316 316 316 F F F 4 080-099 168 168 168 316 316 316 F F F 5 100-119 165 165 165 309 309 309 F F F 6 120-139 168 168 168 316 316 316 F F F 7 140-159 168 168 168 316 316 316 F F F 8 160-179 168 168 168 316 316 316 F F F 9 180-199 168 168 168 316 316 316 F F F 10 200-219 168 168 168 316 316 316 F F F 11 220-239 168 168 168 316 316 316 F F F 12 240-259 168 168 168 316 316 316 F F F 13 260-279 164 164 164 308 308 308 F F F 14 280-299 163 163 163 307 307 307 F F F 15 300-319 165 165 165 309 309 309 F F F 16 320-339 166 166 166 311 311 311 F F F 17 340-359 166 166 166 312 312 312 F F F 18 360-379 163 163 163 307 307 307 F F F 19 380-399 166 166 166 313 313 313 F F F 20 400-419 168 168 168 316 316 316 F F F 21 420-439 168 168 168 316 316 316 F F F 22 440-459 168 168 168 316 316 316 F F F 23 460-479 163 163 163 307 307 307 F F F 24 480-499 167 167 167 315 315 315 F F F 25 500-519 162 162 162 305 305 305 F F F 26 520-539 168 168 168 316 316 316 F F F 27 540-559 168 168 168 316 316 316 F F F 28 560-579 168 168 168 316 316 316 F F F 29 580-599 168 168 168 316 316 316 F F F Max gen: 168, Max flush: 316 Begin loop 85 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.193823232Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.20190575Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.20218307Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.202213645Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.202222564Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.202233932Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817} {"msg":"Crucible 2a7510b3-9df5-45a7-ad31-0b91f8cdc2d0 has session id: e1480c84-59a7-4a13-94c0-34f6631d7883","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.202577382Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Upstairs opts: Upstairs UUID: 2a7510b3-9df5-45a7-ad31-0b91f8cdc2d0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.202621706Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Crucible stats registered with UUID: 2a7510b3-9df5-45a7-ad31-0b91f8cdc2d0","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.202632104Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:27.202641223Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.202816677Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"io task","client":"1","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.202854371Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"io task","client":"0","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.20286511Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"io task","client":"2","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.202973054Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"task":"crutest"} {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.202991182Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817} {"msg":"2a7510b3-9df5-45a7-ad31-0b91f8cdc2d0 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203019317Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203030056Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"client":"0","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203040384Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"client":"1","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203050083Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"client":"2","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203059711Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"io task","client":"2","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.20307236Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"io task","client":"1","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203082228Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"io task","client":"0","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"ds_connection connected from Ok(127.0.0.1:49078)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203200931Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"io task","client":"1","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"ds_connection connected from Ok(127.0.0.1:41367)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203264702Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"io task","client":"0","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"ds_connection connected from Ok(127.0.0.1:34116)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203287998Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"io task","client":"2","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203850796Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"client":"1","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.203877052Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"client":"0","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.20388873Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"client":"2","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.20416647Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204207794Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"[0]R flush_numbers[0..12]: [306, 316, 316, 316, 316, 309, 316, 316, 316, 316, 316, 316]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204262586Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"[0]R generation[0..12]: [162, 168, 168, 168, 168, 165, 168, 168, 168, 168, 168, 168]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204273854Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204284942Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"[1]R flush_numbers[0..12]: [306, 316, 316, 316, 316, 309, 316, 316, 316, 316, 316, 316]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204295571Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"[1]R generation[0..12]: [162, 168, 168, 168, 168, 165, 168, 168, 168, 168, 168, 168]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204305979Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204316718Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"[2]R flush_numbers[0..12]: [306, 316, 316, 316, 316, 309, 316, 316, 316, 316, 316, 316]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204327146Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"[2]R generation[0..12]: [162, 168, 168, 168, 168, 165, 168, 168, 168, 168, 168, 168]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204337545Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204348423Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Max found gen is 169","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204357792Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Generation requested: 170 >= found:169","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.20436748Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Next flush: 317","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204376609Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204385948Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"":"downstairs","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204396776Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204405535Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"client":"0","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204415083Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"client":"1","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204424522Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"client":"2","":"downstairs","session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204434201Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2026-04-15T18:16:27.204446979Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"2a7510b3-9df5-45a7-ad31-0b91f8cdc2d0 is now active with session: e1480c84-59a7-4a13-94c0-34f6631d7883","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204456907Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204465906Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204473415Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817,"session_id":"e1480c84-59a7-4a13-94c0-34f6631d7883"} {"msg":"Activated sub_volume 2a7510b3-9df5-45a7-ad31-0b91f8cdc2d0","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:27.204482324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 18 len 24576 data: 13 11 11 10 8 8 02/30 Read block 473 len 20480 03/30 Write block 64 len 32768 data: 14 16 14 14 13 14 13 15 04/30 Read block 159 len 20480 05/30 Read block 530 len 24576 06/30 Write block 209 len 20480 data: 7 5 4 6 7 07/30 Read block 14 len 8192 08/30 Flush 09/30 Write block 99 len 40960 data: 17 17 17 13 14 11 11 12 13 12 10/30 Flush 11/30 Flush 12/30 Write block 166 len 28672 data: 18 17 15 14 13 14 13 13/30 Write block 456 len 24576 data: 17 17 20 19 18 16 14/30 Read block 362 len 32768 15/30 Read block 171 len 24576 16/30 Write block 222 len 24576 data: 13 17 15 15 15 14 17/30 Write block 532 len 20480 data: 16 20 18 19 16 18/30 Read block 584 len 20480 19/30 Read block 507 len 12288 20/30 Write block 312 len 40960 data: 16 16 13 13 16 17 17 19 16 10 21/30 Write block 310 len 36864 data: 15 18 17 17 14 14 17 18 18 22/30 Read block 36 len 4096 23/30 Read block 159 len 8192 24/30 Read block 316 len 36864 25/30 Write block 79 len 28672 data: 13 12 18 20 17 16 17 26/30 Read block 118 len 16384 27/30 Read block 86 len 20480 28/30 Write block 121 len 20480 data: 12 11 11 13 12 29/30 Write block 62 len 12288 data: 13 11 15 30/30 Write block 346 len 32768 data: 15 14 10 12 12 13 15 15 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Sent false 1001 Acked Read 5 Done Done Sent false 1002 Acked Write 8 Done Done Sent false 1003 Acked Read 5 Done Done Sent false 1004 Acked Read 6 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Read 2 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 10 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Read 6 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 30 34 Done 28 28 0 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:27.214275622Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1817} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 162 317 317 306 F F F <--- 1 020-039 170 170 168 317 317 316 F F F <--- 2 040-059 168 168 168 316 316 316 F F F 3 060-079 170 170 168 317 317 316 T T F <--- 4 080-099 170 170 168 318 318 316 T T F <--- 5 100-119 170 170 165 318 318 309 F F F <--- 6 120-139 168 168 168 316 316 316 T T F <--- 7 140-159 168 168 168 316 316 316 F F F 8 160-179 168 168 168 316 316 316 T T F <--- 9 180-199 168 168 168 316 316 316 F F F 10 200-219 170 170 168 317 317 316 F F F <--- 11 220-239 168 168 168 316 316 316 T T F <--- 12 240-259 168 168 168 316 316 316 F F F 13 260-279 164 164 164 308 308 308 F F F 14 280-299 163 163 163 307 307 307 F F F 15 300-319 165 165 165 309 309 309 T T F <--- 16 320-339 166 166 166 311 311 311 T T F <--- 17 340-359 166 166 166 312 312 312 T T F <--- 18 360-379 163 163 163 307 307 307 F F F 19 380-399 166 166 166 313 313 313 F F F 20 400-419 168 168 168 316 316 316 F F F 21 420-439 168 168 168 316 316 316 F F F 22 440-459 168 168 168 316 316 316 T T F <--- 23 460-479 163 163 163 307 307 307 T T F <--- 24 480-499 167 167 167 315 315 315 F F F 25 500-519 162 162 162 305 305 305 F F F 26 520-539 168 168 168 316 316 316 T T F <--- 27 540-559 168 168 168 316 316 316 F F F 28 560-579 168 168 168 316 316 316 F F F 29 580-599 168 168 168 316 316 316 F F F Max gen: 170, Max flush: 318 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 317 317 317 F F F 1 020-039 170 170 170 317 317 317 F F F 2 040-059 168 168 168 316 316 316 F F F 3 060-079 171 171 171 319 319 319 F F F 4 080-099 171 171 171 319 319 319 F F F 5 100-119 170 170 170 318 318 318 F F F 6 120-139 171 171 171 319 319 319 F F F 7 140-159 168 168 168 316 316 316 F F F 8 160-179 171 171 171 319 319 319 F F F 9 180-199 168 168 168 316 316 316 F F F 10 200-219 170 170 170 317 317 317 F F F 11 220-239 171 171 171 319 319 319 F F F 12 240-259 168 168 168 316 316 316 F F F 13 260-279 164 164 164 308 308 308 F F F 14 280-299 163 163 163 307 307 307 F F F 15 300-319 171 171 171 319 319 319 F F F 16 320-339 171 171 171 319 319 319 F F F 17 340-359 171 171 171 319 319 319 F F F 18 360-379 163 163 163 307 307 307 F F F 19 380-399 166 166 166 313 313 313 F F F 20 400-419 168 168 168 316 316 316 F F F 21 420-439 168 168 168 316 316 316 F F F 22 440-459 171 171 171 319 319 319 F F F 23 460-479 171 171 171 319 319 319 F F F 24 480-499 167 167 167 315 315 315 F F F 25 500-519 162 162 162 305 305 305 F F F 26 520-539 171 171 171 319 319 319 F F F 27 540-559 168 168 168 316 316 316 F F F 28 560-579 168 168 168 316 316 316 F F F 29 580-599 168 168 168 316 316 316 F F F Max gen: 171, Max flush: 319 Begin loop 86 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.477242948Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.485413134Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.485710281Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.485736567Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.485745086Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.485756534Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825} {"msg":"Crucible 716f7c1b-48c1-4e9e-a2b3-ccfaff1123bb has session id: 53b000c4-ca3a-44a7-bd89-489a7911eabd","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486098514Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Upstairs opts: Upstairs UUID: 716f7c1b-48c1-4e9e-a2b3-ccfaff1123bb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486131449Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Crucible stats registered with UUID: 716f7c1b-48c1-4e9e-a2b3-ccfaff1123bb","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486142728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:31.486151916Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486295725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"io task","client":"2","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486318232Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"io task","client":"0","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.48632821Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"io task","client":"1","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486462041Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"task":"crutest"} {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486486227Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825} {"msg":"716f7c1b-48c1-4e9e-a2b3-ccfaff1123bb active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486510924Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486520042Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"client":"0","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486529601Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"client":"1","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.48653918Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"client":"2","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486548758Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"io task","client":"2","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486558367Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"io task","client":"0","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486574694Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"io task","client":"1","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"ds_connection connected from Ok(127.0.0.1:51549)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.48667254Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"io task","client":"0","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"ds_connection connected from Ok(127.0.0.1:51437)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486698096Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"io task","client":"2","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"ds_connection connected from Ok(127.0.0.1:61268)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.486709725Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"io task","client":"1","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.487431379Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"client":"1","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.487456706Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"client":"0","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.487755822Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.48776554Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.487775589Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"client":"2","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"[0]R flush_numbers[0..12]: [317, 317, 316, 319, 319, 318, 319, 316, 319, 316, 317, 319]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488078055Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"[0]R generation[0..12]: [170, 170, 168, 171, 171, 170, 171, 168, 171, 168, 170, 171]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488089463Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488100591Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"[1]R flush_numbers[0..12]: [317, 317, 316, 319, 319, 318, 319, 316, 319, 316, 317, 319]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.48811121Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"[1]R generation[0..12]: [170, 170, 168, 171, 171, 170, 171, 168, 171, 168, 170, 171]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488121808Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488135076Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"[2]R flush_numbers[0..12]: [317, 317, 316, 319, 319, 318, 319, 316, 319, 316, 317, 319]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488145675Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"[2]R generation[0..12]: [170, 170, 168, 171, 171, 170, 171, 168, 171, 168, 170, 171]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488156213Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488166992Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.48817644Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Generation requested: 172 >= found:172","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488186099Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Next flush: 320","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488195308Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488204576Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"":"downstairs","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488214425Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488223194Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"client":"0","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488232972Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"client":"1","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488242611Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"client":"2","":"downstairs","session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488252349Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488261428Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"716f7c1b-48c1-4e9e-a2b3-ccfaff1123bb is now active with session: 53b000c4-ca3a-44a7-bd89-489a7911eabd","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-04-15T18:16:31.488275656Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488285944Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825,"session_id":"53b000c4-ca3a-44a7-bd89-489a7911eabd"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488294943Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825} {"msg":"Activated sub_volume 716f7c1b-48c1-4e9e-a2b3-ccfaff1123bb","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:31.488302852Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 398 len 36864 data: 13 12 12 11 13 12 13 14 14 02/30 Read block 364 len 36864 03/30 Read block 92 len 20480 04/30 Read block 71 len 4096 05/30 Read block 19 len 24576 06/30 Read block 134 len 32768 07/30 Read block 468 len 40960 08/30 Write block 90 len 40960 data: 14 11 9 8 10 13 15 18 18 18 09/30 Read block 150 len 4096 10/30 Flush 11/30 Write block 236 len 12288 data: 8 10 13 12/30 Read block 118 len 36864 13/30 Write block 231 len 8192 data: 15 11 14/30 Write block 8 len 12288 data: 9 10 11 15/30 Read block 522 len 12288 16/30 Read block 78 len 32768 17/30 Write block 347 len 24576 data: 15 11 13 13 14 16 18/30 Write block 298 len 4096 data: 9 19/30 Read block 423 len 12288 20/30 Write block 489 len 32768 data: 8 9 10 8 8 10 11 10 21/30 Write block 369 len 16384 data: 10 9 11 12 22/30 Write block 440 len 12288 data: 16 17 19 23/30 Read block 270 len 28672 24/30 Read block 314 len 24576 25/30 Write block 52 len 28672 data: 13 12 14 15 18 16 14 26/30 Write block 315 len 12288 data: 15 18 19 27/30 Write block 361 len 32768 data: 8 5 6 5 10 11 8 9 28/30 Write block 38 len 36864 data: 19 19 21 22 21 20 17 14 12 29/30 Read block 477 len 16384 30/30 Write block 394 len 36864 data: 12 12 12 11 14 13 13 12 14 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Read 5 Done Done Sent false 1003 Acked Read 1 Done Done Sent false 1004 Acked Read 6 Done Done Sent false 1005 Acked Read 8 Done Done Sent false 1006 Acked Read 10 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Read 1 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 3 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Write 3 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Read 8 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Write 1 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Write 4 Done Done Sent false 1021 Acked Write 3 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 6 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 28 30 Done 29 29 2 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1009)) Some(JobId(1009)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:31.497354049Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1825} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 317 317 317 T T F <--- 1 020-039 170 170 170 317 317 317 T T F <--- 2 040-059 168 168 168 316 316 316 T T F <--- 3 060-079 171 171 171 319 319 319 F F F 4 080-099 172 172 171 320 320 319 F F F <--- 5 100-119 170 170 170 318 318 318 F F F 6 120-139 171 171 171 319 319 319 F F F 7 140-159 168 168 168 316 316 316 F F F 8 160-179 171 171 171 319 319 319 F F F 9 180-199 168 168 168 316 316 316 F F F 10 200-219 170 170 170 317 317 317 F F F 11 220-239 171 171 171 319 319 319 T T F <--- 12 240-259 168 168 168 316 316 316 F F F 13 260-279 164 164 164 308 308 308 F F F 14 280-299 163 163 163 307 307 307 T T F <--- 15 300-319 171 171 171 319 319 319 T T F <--- 16 320-339 171 171 171 319 319 319 F F F 17 340-359 171 171 171 319 319 319 T T F <--- 18 360-379 163 163 163 307 307 307 T T F <--- 19 380-399 172 172 166 320 320 313 T T T <--- 20 400-419 172 172 168 320 320 316 T T T <--- 21 420-439 168 168 168 316 316 316 F F F 22 440-459 171 171 171 319 319 319 T T F <--- 23 460-479 171 171 171 319 319 319 F F F 24 480-499 167 167 167 315 315 315 T T F <--- 25 500-519 162 162 162 305 305 305 F F F 26 520-539 171 171 171 319 319 319 F F F 27 540-559 168 168 168 316 316 316 F F F 28 560-579 168 168 168 316 316 316 F F F 29 580-599 168 168 168 316 316 316 F F F Max gen: 172, Max flush: 320 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 321 321 321 F F F 1 020-039 173 173 173 321 321 321 F F F 2 040-059 173 173 173 321 321 321 F F F 3 060-079 171 171 171 319 319 319 F F F 4 080-099 172 172 172 320 320 320 F F F 5 100-119 170 170 170 318 318 318 F F F 6 120-139 171 171 171 319 319 319 F F F 7 140-159 168 168 168 316 316 316 F F F 8 160-179 171 171 171 319 319 319 F F F 9 180-199 168 168 168 316 316 316 F F F 10 200-219 170 170 170 317 317 317 F F F 11 220-239 173 173 173 321 321 321 F F F 12 240-259 168 168 168 316 316 316 F F F 13 260-279 164 164 164 308 308 308 F F F 14 280-299 173 173 173 321 321 321 F F F 15 300-319 173 173 173 321 321 321 F F F 16 320-339 171 171 171 319 319 319 F F F 17 340-359 173 173 173 321 321 321 F F F 18 360-379 173 173 173 321 321 321 F F F 19 380-399 173 173 173 321 321 321 F F F 20 400-419 173 173 173 321 321 321 F F F 21 420-439 168 168 168 316 316 316 F F F 22 440-459 173 173 173 321 321 321 F F F 23 460-479 171 171 171 319 319 319 F F F 24 480-499 173 173 173 321 321 321 F F F 25 500-519 162 162 162 305 305 305 F F F 26 520-539 171 171 171 319 319 319 F F F 27 540-559 168 168 168 316 316 316 F F F 28 560-579 168 168 168 316 316 316 F F F 29 580-599 168 168 168 316 316 316 F F F Max gen: 173, Max flush: 321 Begin loop 87 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.753229482Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:35.76049577Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.760556441Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.767751419Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768036588Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768066303Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768074712Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.76808622Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833} {"msg":"Crucible 43d26fa0-29aa-4fc0-8e34-23229a25c4ff has session id: d7164a14-cb83-41b4-b38b-4ab41589e612","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768391646Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Upstairs opts: Upstairs UUID: 43d26fa0-29aa-4fc0-8e34-23229a25c4ff, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768434379Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Crucible stats registered with UUID: 43d26fa0-29aa-4fc0-8e34-23229a25c4ff","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768445058Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:35.768456506Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768607114Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"io task","client":"2","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768643099Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"io task","client":"0","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768653967Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"io task","client":"1","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768754493Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"task":"crutest"} {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768778909Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833} {"msg":"43d26fa0-29aa-4fc0-8e34-23229a25c4ff active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768796786Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768805695Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"client":"0","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768815344Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"client":"1","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768824952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"client":"2","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768834521Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"io task","client":"2","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768844469Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"io task","client":"0","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768854378Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"io task","client":"1","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"ds_connection connected from Ok(127.0.0.1:43185)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768921298Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"io task","client":"2","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"ds_connection connected from Ok(127.0.0.1:45400)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768936256Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"io task","client":"1","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"ds_connection connected from Ok(127.0.0.1:42018)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.768946734Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"io task","client":"0","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.769576632Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"client":"1","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.769606458Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"client":"0","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.769670999Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"client":"2","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.769911753Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.769952937Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"[0]R flush_numbers[0..12]: [321, 321, 321, 319, 320, 318, 319, 316, 319, 316, 317, 321]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770008209Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"[0]R generation[0..12]: [173, 173, 173, 171, 172, 170, 171, 168, 171, 168, 170, 173]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770019358Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770030316Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"[1]R flush_numbers[0..12]: [321, 321, 321, 319, 320, 318, 319, 316, 319, 316, 317, 321]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770040935Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"[1]R generation[0..12]: [173, 173, 173, 171, 172, 170, 171, 168, 171, 168, 170, 173]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770051273Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770062001Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"[2]R flush_numbers[0..12]: [321, 321, 321, 319, 320, 318, 319, 316, 319, 316, 317, 321]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.77007244Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"[2]R generation[0..12]: [173, 173, 173, 171, 172, 170, 171, 168, 171, 168, 170, 173]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770082908Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770093657Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770104415Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770113924Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Next flush: 322","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770123113Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770132351Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"":"downstairs","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.7701419Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770154398Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"client":"0","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Transition from WaitQuorum to Active","Wait for a query_work_queue command to finish before sending IO v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770168616Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"client":"1","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770178384Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"client":"2","":"downstairs","session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770188293Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770197492Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"43d26fa0-29aa-4fc0-8e34-23229a25c4ff is now active with session: d7164a14-cb83-41b4-b38b-4ab41589e612","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.7702075Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770216539Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770223928Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833,"session_id":"d7164a14-cb83-41b4-b38b-4ab41589e612"} {"msg":"Activated sub_volume 43d26fa0-29aa-4fc0-8e34-23229a25c4ff","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:35.770232787Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 375 len 28672 data: 14 16 17 13 15 13 8 02/30 Write block 574 len 40960 data: 13 15 18 16 16 15 14 13 12 14 03/30 Write block 411 len 24576 data: 10 9 11 10 10 10 04/30 Write block 357 len 28672 data: 15 13 9 11 9 6 7 05/30 Write block 165 len 8192 data: 21 19 06/30 Read block 431 len 24576 07/30 Read block 120 len 8192 08/30 Read block 344 len 12288 09/30 Write block 563 len 32768 data: 13 14 12 9 10 8 12 13 10/30 Write block 451 len 40960 data: 10 10 10 13 16 18 18 21 20 19 11/30 Write block 329 len 32768 data: 12 15 16 14 11 11 10 11 12/30 Write block 295 len 4096 data: 10 13/30 Write block 312 len 4096 data: 18 14/30 Read block 344 len 32768 15/30 Read block 413 len 4096 16/30 Read block 536 len 16384 17/30 Read block 472 len 20480 18/30 Write block 487 len 8192 data: 9 10 19/30 Flush 20/30 Write block 53 len 28672 data: 13 15 16 19 17 15 13 21/30 Write block 588 len 24576 data: 13 9 11 11 12 11 22/30 Write block 411 len 16384 data: 11 10 12 11 23/30 Read block 37 len 16384 24/30 Read block 468 len 16384 25/30 Read block 432 len 4096 26/30 Read block 378 len 20480 27/30 Read block 569 len 32768 28/30 Read block 365 len 20480 29/30 Write block 209 len 40960 data: 8 6 5 7 8 5 6 8 8 9 30/30 Write block 550 len 36864 data: 11 12 11 12 13 15 16 18 17 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Sent Done Done false 1001 Acked Write 10 Sent Done Done false 1002 Acked Write 6 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Write 2 Sent Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Write 1 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:35.778321674Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1833} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 321 321 321 F F F 1 020-039 173 173 173 321 321 321 F F F 2 040-059 173 173 173 321 321 321 F T T <--- 3 060-079 171 171 171 319 319 319 F F F 4 080-099 172 172 172 320 320 320 F F F 5 100-119 170 170 170 318 318 318 F F F 6 120-139 171 171 171 319 319 319 F F F 7 140-159 168 168 168 316 316 316 F F F 8 160-179 171 174 174 319 322 322 F F F <--- 9 180-199 168 168 168 316 316 316 F F F 10 200-219 170 170 170 317 317 317 F T T <--- 11 220-239 173 173 173 321 321 321 F F F 12 240-259 168 168 168 316 316 316 F F F 13 260-279 164 164 164 308 308 308 F F F 14 280-299 173 174 174 321 322 322 F F F <--- 15 300-319 173 174 174 321 322 322 F F F <--- 16 320-339 171 174 174 319 322 322 F F F <--- 17 340-359 173 174 174 321 322 322 F F F <--- 18 360-379 173 174 174 321 322 322 F F F <--- 19 380-399 173 174 174 321 322 322 F F F <--- 20 400-419 173 174 174 321 322 322 F T T <--- 21 420-439 168 168 168 316 316 316 F F F 22 440-459 173 174 174 321 322 322 F F F <--- 23 460-479 171 174 174 319 322 322 F F F <--- 24 480-499 173 174 174 321 322 322 F F F <--- 25 500-519 162 162 162 305 305 305 F F F 26 520-539 171 171 171 319 319 319 F F F 27 540-559 168 168 168 316 316 316 F T T <--- 28 560-579 168 174 174 316 322 322 F F F <--- 29 580-599 168 174 174 316 322 322 F T T <--- Max gen: 174, Max flush: 322 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 321 321 321 F F F 1 020-039 173 173 173 321 321 321 F F F 2 040-059 175 175 175 323 323 323 F F F 3 060-079 171 171 171 319 319 319 F F F 4 080-099 172 172 172 320 320 320 F F F 5 100-119 170 170 170 318 318 318 F F F 6 120-139 171 171 171 319 319 319 F F F 7 140-159 168 168 168 316 316 316 F F F 8 160-179 174 174 174 322 322 322 F F F 9 180-199 168 168 168 316 316 316 F F F 10 200-219 175 175 175 323 323 323 F F F 11 220-239 173 173 173 321 321 321 F F F 12 240-259 168 168 168 316 316 316 F F F 13 260-279 164 164 164 308 308 308 F F F 14 280-299 174 174 174 322 322 322 F F F 15 300-319 174 174 174 322 322 322 F F F 16 320-339 174 174 174 322 322 322 F F F 17 340-359 174 174 174 322 322 322 F F F 18 360-379 174 174 174 322 322 322 F F F 19 380-399 174 174 174 322 322 322 F F F 20 400-419 175 175 175 323 323 323 F F F 21 420-439 168 168 168 316 316 316 F F F 22 440-459 174 174 174 322 322 322 F F F 23 460-479 174 174 174 322 322 322 F F F 24 480-499 174 174 174 322 322 322 F F F 25 500-519 162 162 162 305 305 305 F F F 26 520-539 171 171 171 319 319 319 F F F 27 540-559 175 175 175 323 323 323 F F F 28 560-579 174 174 174 322 322 322 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 175, Max flush: 323 Begin loop 88 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.068317139Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.076612576Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.076903364Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.076939738Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.076948667Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.076960185Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841} {"msg":"Crucible 85a49442-b78e-4b33-ad13-df769d151bf8 has session id: eef80e26-e1d2-4c9c-8788-29b5695127d3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077311684Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Upstairs opts: Upstairs UUID: 85a49442-b78e-4b33-ad13-df769d151bf8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.0773418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Crucible stats registered with UUID: 85a49442-b78e-4b33-ad13-df769d151bf8","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077354248Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:40.077363216Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077525833Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"io task","client":"2","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077564957Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"io task","client":"0","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077575765Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"io task","client":"1","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.07768169Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"task":"crutest"} {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077699197Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841} {"msg":"85a49442-b78e-4b33-ad13-df769d151bf8 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077722484Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077731773Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"client":"0","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077743421Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"client":"1","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077766907Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"client":"2","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077776336Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"io task","client":"2","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077806542Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"io task","client":"0","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.07782016Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"io task","client":"1","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"ds_connection connected from Ok(127.0.0.1:42318)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077934113Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"io task","client":"2","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"ds_connection connected from Ok(127.0.0.1:47355)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.077946111Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"io task","client":"0","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"ds_connection connected from Ok(127.0.0.1:58116)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.07795641Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"io task","client":"1","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.078552803Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"client":"2","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.078565671Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"client":"0","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.078631091Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"client":"1","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.078880015Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.078889403Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"[0]R flush_numbers[0..12]: [321, 321, 323, 319, 320, 318, 319, 316, 322, 316, 323, 321]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.078984989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"[0]R generation[0..12]: [173, 173, 175, 171, 172, 170, 171, 168, 174, 168, 175, 173]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.078996238Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079008646Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"[1]R flush_numbers[0..12]: [321, 321, 323, 319, 320, 318, 319, 316, 322, 316, 323, 321]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079019244Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"[1]R generation[0..12]: [173, 173, 175, 171, 172, 170, 171, 168, 174, 168, 175, 173]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079029623Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079040401Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"[2]R flush_numbers[0..12]: [321, 321, 323, 319, 320, 318, 319, 316, 322, 316, 323, 321]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.07905102Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"[2]R generation[0..12]: [173, 173, 175, 171, 172, 170, 171, 168, 174, 168, 175, 173]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079061448Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079072327Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Max found gen is 176","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079081755Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079091424Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Next flush: 324","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079100822Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079109981Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"":"downstairs","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.07911989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079128728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"client":"0","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079138277Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"client":"1","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079148865Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"client":"2","":"downstairs","session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079158674Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-04-15T18:16:40.079167643Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"85a49442-b78e-4b33-ad13-df769d151bf8 is now active with session: eef80e26-e1d2-4c9c-8788-29b5695127d3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079182001Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079190819Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079198358Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841,"session_id":"eef80e26-e1d2-4c9c-8788-29b5695127d3"} {"msg":"Activated sub_volume 85a49442-b78e-4b33-ad13-df769d151bf8","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:40.079207417Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 261 len 12288 data: 11 14 16 02/30 Flush 03/30 Write block 273 len 36864 data: 16 13 8 9 9 7 14 13 14 04/30 Read block 386 len 12288 05/30 Write block 437 len 12288 data: 22 23 18 06/30 Write block 244 len 24576 data: 17 16 17 15 13 13 07/30 Write block 297 len 40960 data: 8 10 8 6 7 7 8 10 11 14 08/30 Read block 509 len 40960 09/30 Read block 274 len 24576 10/30 Read block 243 len 40960 11/30 Read block 16 len 36864 12/30 Read block 468 len 8192 13/30 Write block 516 len 28672 data: 12 13 11 12 11 13 13 14/30 Write block 145 len 4096 data: 16 15/30 Read block 192 len 8192 16/30 Read block 245 len 24576 17/30 Read block 162 len 28672 18/30 Read block 532 len 28672 19/30 Write block 81 len 32768 data: 19 21 18 17 18 15 16 16 20/30 Read block 374 len 8192 21/30 Read block 353 len 36864 22/30 Read block 32 len 40960 23/30 Read block 447 len 20480 24/30 Read block 91 len 12288 25/30 Write block 514 len 20480 data: 15 13 13 14 12 26/30 Read block 329 len 32768 27/30 Read block 19 len 20480 28/30 Write block 307 len 8192 data: 11 12 29/30 Write block 147 len 16384 data: 11 10 10 10 30/30 Read block 183 len 36864 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 9 Done Done Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Write 6 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Read 2 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Read 2 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Read 7 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Write 4 Done Sent Sent false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 2 29 Done 28 1 26 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:40.088232258Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1841} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 321 321 321 F F F 1 020-039 173 173 173 321 321 321 F F F 2 040-059 175 175 175 323 323 323 F F F 3 060-079 171 171 171 319 319 319 F F F 4 080-099 172 172 172 320 320 320 T F T <--- 5 100-119 170 170 170 318 318 318 F F F 6 120-139 171 171 171 319 319 319 F F F 7 140-159 168 168 168 316 316 316 T F T <--- 8 160-179 174 174 174 322 322 322 F F F 9 180-199 168 168 168 316 316 316 F F F 10 200-219 175 175 175 323 323 323 F F F 11 220-239 173 173 173 321 321 321 F F F 12 240-259 168 168 168 316 316 316 T F T <--- 13 260-279 176 176 176 324 324 324 T T T 14 280-299 174 174 174 322 322 322 T T T 15 300-319 174 174 174 322 322 322 T F T <--- 16 320-339 174 174 174 322 322 322 F F F 17 340-359 174 174 174 322 322 322 F F F 18 360-379 174 174 174 322 322 322 F F F 19 380-399 174 174 174 322 322 322 F F F 20 400-419 175 175 175 323 323 323 F F F 21 420-439 168 168 168 316 316 316 T F T <--- 22 440-459 174 174 174 322 322 322 F F F 23 460-479 174 174 174 322 322 322 F F F 24 480-499 174 174 174 322 322 322 F F F 25 500-519 162 162 162 305 305 305 T F T <--- 26 520-539 171 171 171 319 319 319 T F T <--- 27 540-559 175 175 175 323 323 323 F F F 28 560-579 174 174 174 322 322 322 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 176, Max flush: 324 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 321 321 321 F F F 1 020-039 173 173 173 321 321 321 F F F 2 040-059 175 175 175 323 323 323 F F F 3 060-079 171 171 171 319 319 319 F F F 4 080-099 177 177 177 325 325 325 F F F 5 100-119 170 170 170 318 318 318 F F F 6 120-139 171 171 171 319 319 319 F F F 7 140-159 177 177 177 325 325 325 F F F 8 160-179 174 174 174 322 322 322 F F F 9 180-199 168 168 168 316 316 316 F F F 10 200-219 175 175 175 323 323 323 F F F 11 220-239 173 173 173 321 321 321 F F F 12 240-259 177 177 177 325 325 325 F F F 13 260-279 177 177 177 325 325 325 F F F 14 280-299 177 177 177 325 325 325 F F F 15 300-319 177 177 177 325 325 325 F F F 16 320-339 174 174 174 322 322 322 F F F 17 340-359 174 174 174 322 322 322 F F F 18 360-379 174 174 174 322 322 322 F F F 19 380-399 174 174 174 322 322 322 F F F 20 400-419 175 175 175 323 323 323 F F F 21 420-439 177 177 177 325 325 325 F F F 22 440-459 174 174 174 322 322 322 F F F 23 460-479 174 174 174 322 322 322 F F F 24 480-499 174 174 174 322 322 322 F F F 25 500-519 177 177 177 325 325 325 F F F 26 520-539 177 177 177 325 325 325 F F F 27 540-559 175 175 175 323 323 323 F F F 28 560-579 174 174 174 322 322 322 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 177, Max flush: 325 Begin loop 89 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.335929076Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.344048549Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.344337377Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.344363103Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.344371832Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.34438332Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849} {"msg":"Crucible 214b0309-33f4-4718-b097-6ca0d16a26b5 has session id: d28cc387-7b89-4e3e-b1c5-e53684cb5c99","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.344732769Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Upstairs opts: Upstairs UUID: 214b0309-33f4-4718-b097-6ca0d16a26b5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.344761775Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Crucible stats registered with UUID: 214b0309-33f4-4718-b097-6ca0d16a26b5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.344775243Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:44.344784362Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.344940539Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"io task","client":"2","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.344977923Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"io task","client":"1","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.344988942Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"io task","client":"0","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.34507194Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"task":"crutest"} {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345092157Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849} {"msg":"214b0309-33f4-4718-b097-6ca0d16a26b5 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345120853Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345130311Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"client":"0","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.34514064Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"client":"1","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345150378Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"client":"2","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345159947Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"io task","client":"2","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345169855Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"io task","client":"0","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345179904Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"io task","client":"1","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"ds_connection connected from Ok(127.0.0.1:58872)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345254993Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"io task","client":"0","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"ds_connection connected from Ok(127.0.0.1:56344)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345280269Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"io task","client":"2","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"ds_connection connected from Ok(127.0.0.1:37846)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345388363Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"io task","client":"1","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345915126Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"client":"2","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.345979387Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"client":"0","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346044958Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"client":"1","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346206494Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346224081Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"[0]R flush_numbers[0..12]: [321, 321, 323, 319, 325, 318, 319, 325, 322, 316, 323, 321]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346327106Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"[0]R generation[0..12]: [173, 173, 175, 171, 177, 170, 171, 177, 174, 168, 175, 173]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346338375Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346349333Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"[1]R flush_numbers[0..12]: [321, 321, 323, 319, 325, 318, 319, 325, 322, 316, 323, 321]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346359831Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"[1]R generation[0..12]: [173, 173, 175, 171, 177, 170, 171, 177, 174, 168, 175, 173]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.34637014Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346381098Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"[2]R flush_numbers[0..12]: [321, 321, 323, 319, 325, 318, 319, 325, 322, 316, 323, 321]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346391767Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"[2]R generation[0..12]: [173, 173, 175, 171, 177, 170, 171, 177, 174, 168, 175, 173]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346402225Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346415473Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346424892Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346434481Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Next flush: 326","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346443729Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346452948Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"":"downstairs","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346462626Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346471275Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"client":"0","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346485303Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"client":"1","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346494752Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"client":"2","":"downstairs","session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.34650443Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346513429Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"214b0309-33f4-4718-b097-6ca0d16a26b5 is now active with session: d28cc387-7b89-4e3e-b1c5-e53684cb5c99","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346523248Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346532296Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346539815Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849,"session_id":"d28cc387-7b89-4e3e-b1c5-e53684cb5c99"} {"msg":"Activated sub_volume 214b0309-33f4-4718-b097-6ca0d16a26b5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:44.346548804Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 42 len 40960 data: 22 21 18 15 13 15 13 14 11 11 03/30 Read block 450 len 4096 04/30 Write block 349 len 24576 data: 14 14 15 17 16 15 05/30 Write block 347 len 40960 data: 16 12 15 15 16 18 17 16 16 16 06/30 Read block 374 len 8192 07/30 Read block 61 len 28672 08/30 Read block 58 len 36864 09/30 Write block 230 len 32768 data: 18 16 12 11 9 8 9 11 10/30 Read block 94 len 36864 11/30 Read block 379 len 32768 12/30 Read block 222 len 4096 13/30 Write block 17 len 4096 data: 15 14/30 Write block 362 len 32768 data: 7 8 6 11 12 9 10 11 15/30 Write block 469 len 36864 data: 16 17 17 17 19 18 15 10 8 16/30 Write block 140 len 4096 data: 21 17/30 Read block 498 len 24576 18/30 Write block 51 len 12288 data: 12 14 14 19/30 Write block 344 len 28672 data: 13 16 16 17 13 16 16 20/30 Write block 514 len 20480 data: 16 14 14 15 13 21/30 Read block 491 len 24576 22/30 Write block 168 len 40960 data: 16 15 14 15 14 13 13 12 15 16 23/30 Read block 446 len 36864 24/30 Read block 461 len 28672 25/30 Write block 559 len 40960 data: 16 16 16 14 14 15 13 10 11 9 26/30 Write block 259 len 32768 data: 8 6 12 15 17 17 18 20 27/30 Write block 456 len 16384 data: 19 19 22 21 28/30 Write block 9 len 32768 data: 11 12 13 14 15 16 15 14 29/30 Read block 307 len 36864 30/30 Read block 140 len 40960 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 10 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 6 Done Done Done false 1004 Acked Write 10 Done Done Done false 1005 Acked Read 2 Done Done Done false 1006 Acked Read 7 Done Done Done false 1007 Acked Read 9 Done Done Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 1 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Write 3 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 8 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Read 10 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 22 0 22 Done 29 7 29 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:44.355638385Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1849} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 321 321 321 T F T <--- 1 020-039 173 173 173 321 321 321 F F F 2 040-059 175 175 175 323 323 323 T T T 3 060-079 171 171 171 319 319 319 F F F 4 080-099 177 177 177 325 325 325 F F F 5 100-119 170 170 170 318 318 318 F F F 6 120-139 171 171 171 319 319 319 F F F 7 140-159 177 177 177 325 325 325 T F T <--- 8 160-179 174 174 174 322 322 322 T F T <--- 9 180-199 168 168 168 316 316 316 F F F 10 200-219 175 175 175 323 323 323 F F F 11 220-239 173 173 173 321 321 321 T F T <--- 12 240-259 177 177 177 325 325 325 T F T <--- 13 260-279 177 177 177 325 325 325 T F T <--- 14 280-299 177 177 177 325 325 325 F F F 15 300-319 177 177 177 325 325 325 F F F 16 320-339 174 174 174 322 322 322 F F F 17 340-359 174 174 174 322 322 322 T T T 18 360-379 174 174 174 322 322 322 T F T <--- 19 380-399 174 174 174 322 322 322 F F F 20 400-419 175 175 175 323 323 323 F F F 21 420-439 177 177 177 325 325 325 F F F 22 440-459 174 174 174 322 322 322 T F T <--- 23 460-479 174 174 174 322 322 322 T F T <--- 24 480-499 174 174 174 322 322 322 F F F 25 500-519 177 177 177 325 325 325 T F T <--- 26 520-539 177 177 177 325 325 325 F F F 27 540-559 175 175 175 323 323 323 T F T <--- 28 560-579 174 174 174 322 322 322 T F T <--- 29 580-599 175 175 175 323 323 323 F F F Max gen: 177, Max flush: 325 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 326 326 326 F F F 1 020-039 173 173 173 321 321 321 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 171 171 171 319 319 319 F F F 4 080-099 177 177 177 325 325 325 F F F 5 100-119 170 170 170 318 318 318 F F F 6 120-139 171 171 171 319 319 319 F F F 7 140-159 178 178 178 326 326 326 F F F 8 160-179 178 178 178 326 326 326 F F F 9 180-199 168 168 168 316 316 316 F F F 10 200-219 175 175 175 323 323 323 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 178 178 178 326 326 326 F F F 13 260-279 178 178 178 326 326 326 F F F 14 280-299 177 177 177 325 325 325 F F F 15 300-319 177 177 177 325 325 325 F F F 16 320-339 174 174 174 322 322 322 F F F 17 340-359 178 178 178 326 326 326 F F F 18 360-379 178 178 178 326 326 326 F F F 19 380-399 174 174 174 322 322 322 F F F 20 400-419 175 175 175 323 323 323 F F F 21 420-439 177 177 177 325 325 325 F F F 22 440-459 178 178 178 326 326 326 F F F 23 460-479 178 178 178 326 326 326 F F F 24 480-499 174 174 174 322 322 322 F F F 25 500-519 178 178 178 326 326 326 F F F 26 520-539 177 177 177 325 325 325 F F F 27 540-559 178 178 178 326 326 326 F F F 28 560-579 178 178 178 326 326 326 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 178, Max flush: 326 Begin loop 90 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.61581411Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.623892249Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624217511Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624252236Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624261335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624272943Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857} {"msg":"Crucible cd04960a-3856-45df-b5f2-d670a3b7e477 has session id: 31d62182-bdee-46fd-9d53-51fc1e4b2d8a","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624562211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Upstairs opts: Upstairs UUID: cd04960a-3856-45df-b5f2-d670a3b7e477, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624593796Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Crucible stats registered with UUID: cd04960a-3856-45df-b5f2-d670a3b7e477","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624604355Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:48.624613273Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624768301Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"io task","client":"1","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624805025Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"io task","client":"0","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624815864Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"io task","client":"2","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624914459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"task":"crutest"} {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624939686Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857} {"msg":"cd04960a-3856-45df-b5f2-d670a3b7e477 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624957823Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624966652Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"client":"0","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.62497611Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"client":"1","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624985749Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"client":"2","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.624995298Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"io task","client":"2","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.625005456Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"io task","client":"0","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.625015195Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"io task","client":"1","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"ds_connection connected from Ok(127.0.0.1:38779)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.625083285Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"io task","client":"1","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"ds_connection connected from Ok(127.0.0.1:46428)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.625108431Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"io task","client":"2","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"ds_connection connected from Ok(127.0.0.1:53582)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.625119759Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"io task","client":"0","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.625707324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"client":"1","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.62573124Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"client":"0","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.625828486Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"client":"2","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.625976214Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.625991442Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"[0]R flush_numbers[0..12]: [326, 321, 326, 319, 325, 318, 319, 326, 326, 316, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626084798Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"[0]R generation[0..12]: [178, 173, 178, 171, 177, 170, 171, 178, 178, 168, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626096297Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626107215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"[1]R flush_numbers[0..12]: [326, 321, 326, 319, 325, 318, 319, 326, 326, 316, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626117664Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"[1]R generation[0..12]: [178, 173, 178, 171, 177, 170, 171, 178, 178, 168, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626128172Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.62613898Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"[2]R flush_numbers[0..12]: [326, 321, 326, 319, 325, 318, 319, 326, 326, 316, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626149639Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"[2]R generation[0..12]: [178, 173, 178, 171, 177, 170, 171, 178, 178, 168, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626160167Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626170916Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Max found gen is 179","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626180304Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Generation requested: 180 >= found:179","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626189883Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Next flush: 327","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626199132Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.62620845Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"":"downstairs","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626218069Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626227778Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"client":"0","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626241515Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"client":"1","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626251054Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"client":"2","":"downstairs","session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626260663Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626269621Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"cd04960a-3856-45df-b5f2-d670a3b7e477 is now active with session: 31d62182-bdee-46fd-9d53-51fc1e4b2d8a","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.62627949Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626288599Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626296008Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857,"session_id":"31d62182-bdee-46fd-9d53-51fc1e4b2d8a"} {"msg":"Activated sub_volume cd04960a-3856-45df-b5f2-d670a3b7e477","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:48.626305126Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 342 len 36864 data: 15 14 14 17 17 18 14 17 17 02/30 Read block 595 len 8192 03/30 Read block 561 len 24576 04/30 Read block 415 len 4096 05/30 Read block 555 len 12288 06/30 Write block 70 len 36864 data: 14 16 16 12 12 11 12 11 13 07/30 Read block 557 len 4096 08/30 Write block 22 len 20480 data: 9 9 9 13 13 09/30 Read block 134 len 12288 10/30 Write block 303 len 28672 data: 9 11 12 15 12 13 15 11/30 Read block 437 len 12288 12/30 Read block 449 len 36864 13/30 Write block 558 len 28672 data: 18 17 17 17 15 15 16 14/30 Flush 15/30 Write block 87 len 16384 data: 17 17 17 15 16/30 Read block 450 len 8192 17/30 Read block 335 len 8192 18/30 Read block 472 len 8192 19/30 Read block 327 len 36864 20/30 Read block 186 len 32768 21/30 Flush 22/30 Write block 384 len 4096 data: 9 23/30 Read block 97 len 40960 24/30 Write block 101 len 12288 data: 18 14 15 25/30 Write block 175 len 40960 data: 13 16 17 18 17 21 20 20 20 16 26/30 Write block 520 len 12288 data: 12 14 14 27/30 Write block 133 len 8192 data: 14 14 28/30 Read block 370 len 12288 29/30 Write block 271 len 8192 data: 17 16 30/30 Write block 177 len 8192 data: 18 19 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Read 1 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Write 9 Done Done Done false 1006 Acked Read 1 Done Done Done false 1007 Acked Write 5 Done Done Done false 1008 Acked Read 3 Done Done Done false 1009 Acked Write 7 Done Done Done false 1010 Acked Read 3 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Write 7 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Read 8 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Write 3 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 20 24 Done 28 28 10 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:48.635109109Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1857} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 326 326 326 F F F 1 020-039 180 180 173 327 327 321 F F T <--- 2 040-059 178 178 178 326 326 326 F F F 3 060-079 180 180 171 327 327 319 F F T <--- 4 080-099 180 180 177 328 328 325 F F F <--- 5 100-119 170 170 170 318 318 318 T T F <--- 6 120-139 171 171 171 319 319 319 T T F <--- 7 140-159 178 178 178 326 326 326 F F F 8 160-179 178 178 178 326 326 326 T T F <--- 9 180-199 168 168 168 316 316 316 T T F <--- 10 200-219 175 175 175 323 323 323 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 178 178 178 326 326 326 F F F 13 260-279 178 178 178 326 326 326 T T F <--- 14 280-299 177 177 177 325 325 325 F F F 15 300-319 180 180 177 327 327 325 F F T <--- 16 320-339 174 174 174 322 322 322 F F F 17 340-359 180 180 178 327 327 326 F F T <--- 18 360-379 178 178 178 326 326 326 F F F 19 380-399 174 174 174 322 322 322 T T F <--- 20 400-419 175 175 175 323 323 323 F F F 21 420-439 177 177 177 325 325 325 F F F 22 440-459 178 178 178 326 326 326 F F F 23 460-479 178 178 178 326 326 326 F F F 24 480-499 174 174 174 322 322 322 F F F 25 500-519 178 178 178 326 326 326 F F F 26 520-539 177 177 177 325 325 325 T T F <--- 27 540-559 180 180 178 327 327 326 F F F <--- 28 560-579 180 180 178 327 327 326 F F F <--- 29 580-599 175 175 175 323 323 323 F F F Max gen: 180, Max flush: 328 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 326 326 326 F F F 1 020-039 180 180 180 327 327 327 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 180 180 180 327 327 327 F F F 4 080-099 180 180 180 328 328 328 F F F 5 100-119 181 181 181 329 329 329 F F F 6 120-139 181 181 181 329 329 329 F F F 7 140-159 178 178 178 326 326 326 F F F 8 160-179 181 181 181 329 329 329 F F F 9 180-199 181 181 181 329 329 329 F F F 10 200-219 175 175 175 323 323 323 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 178 178 178 326 326 326 F F F 13 260-279 181 181 181 329 329 329 F F F 14 280-299 177 177 177 325 325 325 F F F 15 300-319 180 180 180 327 327 327 F F F 16 320-339 174 174 174 322 322 322 F F F 17 340-359 180 180 180 327 327 327 F F F 18 360-379 178 178 178 326 326 326 F F F 19 380-399 181 181 181 329 329 329 F F F 20 400-419 175 175 175 323 323 323 F F F 21 420-439 177 177 177 325 325 325 F F F 22 440-459 178 178 178 326 326 326 F F F 23 460-479 178 178 178 326 326 326 F F F 24 480-499 174 174 174 322 322 322 F F F 25 500-519 178 178 178 326 326 326 F F F 26 520-539 181 181 181 329 329 329 F F F 27 540-559 180 180 180 327 327 327 F F F 28 560-579 180 180 180 327 327 327 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 181, Max flush: 329 Begin loop 91 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.893205207Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.901334109Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.901636495Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.901664771Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.901673659Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.901685518Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865} {"msg":"Crucible 575a2dbe-8d80-4185-b9a7-d559e3da6c02 has session id: 69e822b8-ec34-4051-8d7b-3daddea831ad","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.901977395Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Upstairs opts: Upstairs UUID: 575a2dbe-8d80-4185-b9a7-d559e3da6c02, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902022678Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Crucible stats registered with UUID: 575a2dbe-8d80-4185-b9a7-d559e3da6c02","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902033137Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:52.902042116Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902202162Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"io task","client":"1","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902239317Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"io task","client":"0","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902250265Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"io task","client":"2","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902344341Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"task":"crutest"} {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902360579Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865} {"msg":"575a2dbe-8d80-4185-b9a7-d559e3da6c02 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902396094Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902405632Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"client":"0","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902415431Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"client":"1","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.90242501Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"client":"2","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902434708Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"io task","client":"2","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902444587Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"io task","client":"0","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902456995Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"io task","client":"1","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:35400)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902484631Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"io task","client":"2","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:46138)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902495619Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"io task","client":"0","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:53254)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.902515906Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"io task","client":"1","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903199206Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"client":"1","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903214234Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"client":"2","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903224763Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"client":"0","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903555874Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903565263Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"[0]R flush_numbers[0..12]: [326, 327, 326, 327, 328, 329, 329, 326, 329, 329, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903603117Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"[0]R generation[0..12]: [178, 180, 178, 180, 180, 181, 181, 178, 181, 181, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903614216Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903624974Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"[1]R flush_numbers[0..12]: [326, 327, 326, 327, 328, 329, 329, 326, 329, 329, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903635523Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"[1]R generation[0..12]: [178, 180, 178, 180, 180, 181, 181, 178, 181, 181, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903645831Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.90365663Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"[2]R flush_numbers[0..12]: [326, 327, 326, 327, 328, 329, 329, 326, 329, 329, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903668788Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"[2]R generation[0..12]: [178, 180, 178, 180, 180, 181, 181, 178, 181, 181, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903679126Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903689795Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Max found gen is 182","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903699163Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903708612Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Next flush: 330","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903717721Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903727189Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"":"downstairs","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903736718Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903745397Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"client":"0","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903754985Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"client":"1","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-04-15T18:16:52.903764474Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"client":"2","":"downstairs","session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903777172Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903786171Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"575a2dbe-8d80-4185-b9a7-d559e3da6c02 is now active with session: 69e822b8-ec34-4051-8d7b-3daddea831ad","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903796159Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903805168Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903812637Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865,"session_id":"69e822b8-ec34-4051-8d7b-3daddea831ad"} {"msg":"Activated sub_volume 575a2dbe-8d80-4185-b9a7-d559e3da6c02","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:52.903822915Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 574 len 4096 data: 14 02/30 Read block 218 len 40960 03/30 Read block 333 len 28672 04/30 Read block 357 len 40960 05/30 Write block 75 len 20480 data: 12 13 12 14 14 06/30 Read block 390 len 20480 07/30 Read block 157 len 24576 08/30 Write block 173 len 32768 data: 14 14 14 17 19 20 18 22 09/30 Flush 10/30 Write block 499 len 28672 data: 13 11 10 9 7 7 12 11/30 Read block 78 len 16384 12/30 Read block 160 len 28672 13/30 Read block 41 len 24576 14/30 Write block 275 len 32768 data: 9 10 10 8 15 14 15 11 15/30 Write block 264 len 8192 data: 18 19 16/30 Flush 17/30 Write block 556 len 28672 data: 17 19 19 18 18 18 16 18/30 Read block 380 len 16384 19/30 Flush 20/30 Write block 333 len 8192 data: 12 12 21/30 Write block 570 len 32768 data: 14 14 15 12 15 16 19 17 22/30 Read block 86 len 24576 23/30 Read block 212 len 28672 24/30 Read block 165 len 8192 25/30 Read block 396 len 24576 26/30 Read block 533 len 12288 27/30 Read block 225 len 8192 28/30 Read block 428 len 8192 29/30 Read block 498 len 16384 30/30 Read block 324 len 8192 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Read 10 Done Done Done false 1004 Acked Write 5 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Read 6 Done Sent Done false 1007 Acked Write 8 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 7 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 2 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 7 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Read 2 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 26 0 26 Done 30 4 30 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) None Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:52.913426502Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1865} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 326 326 326 F F F 1 020-039 180 180 180 327 327 327 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 182 180 182 330 327 330 F F F <--- 4 080-099 180 180 180 328 328 328 F F F 5 100-119 181 181 181 329 329 329 F F F 6 120-139 181 181 181 329 329 329 F F F 7 140-159 178 178 178 326 326 326 F F F 8 160-179 182 181 182 330 329 330 F F F <--- 9 180-199 182 181 182 330 329 330 F F F <--- 10 200-219 175 175 175 323 323 323 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 178 178 178 326 326 326 F F F 13 260-279 182 181 182 331 329 331 F F F <--- 14 280-299 182 177 182 331 325 331 F F F <--- 15 300-319 180 180 180 327 327 327 F F F 16 320-339 174 174 174 322 322 322 T F T <--- 17 340-359 180 180 180 327 327 327 F F F 18 360-379 178 178 178 326 326 326 F F F 19 380-399 181 181 181 329 329 329 F F F 20 400-419 175 175 175 323 323 323 F F F 21 420-439 177 177 177 325 325 325 F F F 22 440-459 178 178 178 326 326 326 F F F 23 460-479 178 178 178 326 326 326 F F F 24 480-499 182 174 182 331 322 331 F F F <--- 25 500-519 182 178 182 331 326 331 F F F <--- 26 520-539 181 181 181 329 329 329 F F F 27 540-559 182 180 182 332 327 332 F F F <--- 28 560-579 182 180 182 332 327 332 T T T <--- 29 580-599 175 175 175 323 323 323 F F F Max gen: 182, Max flush: 332 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 326 326 326 F F F 1 020-039 180 180 180 327 327 327 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 182 182 182 330 330 330 F F F 4 080-099 180 180 180 328 328 328 F F F 5 100-119 181 181 181 329 329 329 F F F 6 120-139 181 181 181 329 329 329 F F F 7 140-159 178 178 178 326 326 326 F F F 8 160-179 182 182 182 330 330 330 F F F 9 180-199 182 182 182 330 330 330 F F F 10 200-219 175 175 175 323 323 323 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 178 178 178 326 326 326 F F F 13 260-279 182 182 182 331 331 331 F F F 14 280-299 182 182 182 331 331 331 F F F 15 300-319 180 180 180 327 327 327 F F F 16 320-339 183 183 183 333 333 333 F F F 17 340-359 180 180 180 327 327 327 F F F 18 360-379 178 178 178 326 326 326 F F F 19 380-399 181 181 181 329 329 329 F F F 20 400-419 175 175 175 323 323 323 F F F 21 420-439 177 177 177 325 325 325 F F F 22 440-459 178 178 178 326 326 326 F F F 23 460-479 178 178 178 326 326 326 F F F 24 480-499 182 182 182 331 331 331 F F F 25 500-519 182 182 182 331 331 331 F F F 26 520-539 181 181 181 329 329 329 F F F 27 540-559 182 182 182 332 332 332 F F F 28 560-579 183 183 183 333 333 333 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 183, Max flush: 333 Begin loop 92 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.161044571Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.169320772Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.169628017Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.169653233Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.169661512Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.16967293Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873} {"msg":"Crucible ae423c0a-6ca3-4cbe-9c60-d8c7daa6e0aa has session id: ba2f7627-8259-4b99-a2fd-74ebd82dde65","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.169980225Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Upstairs opts: Upstairs UUID: ae423c0a-6ca3-4cbe-9c60-d8c7daa6e0aa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170009181Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Crucible stats registered with UUID: ae423c0a-6ca3-4cbe-9c60-d8c7daa6e0aa","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170020379Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:57.170029708Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170180376Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"io task","client":"0","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170202863Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"io task","client":"2","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170212701Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"io task","client":"1","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170354501Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"task":"crutest"} {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170371858Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873} {"msg":"ae423c0a-6ca3-4cbe-9c60-d8c7daa6e0aa active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170395125Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170404323Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"client":"0","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170413892Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"client":"1","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.17042345Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"client":"2","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170433099Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"io task","client":"2","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170442988Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"io task","client":"0","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170452646Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"io task","client":"1","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"ds_connection connected from Ok(127.0.0.1:46571)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170537864Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"io task","client":"1","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"ds_connection connected from Ok(127.0.0.1:43307)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170548782Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"io task","client":"0","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"ds_connection connected from Ok(127.0.0.1:63599)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.170558961Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"io task","client":"2","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171243411Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"client":"0","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171280385Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"client":"1","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171327698Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"client":"2","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171553665Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.17159058Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"[0]R flush_numbers[0..12]: [326, 327, 326, 330, 328, 329, 329, 326, 330, 330, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171615906Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"[0]R generation[0..12]: [178, 180, 178, 182, 180, 181, 181, 178, 182, 182, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171627055Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171637913Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"[1]R flush_numbers[0..12]: [326, 327, 326, 330, 328, 329, 329, 326, 330, 330, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171648371Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"[1]R generation[0..12]: [178, 180, 178, 182, 180, 181, 181, 178, 182, 182, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.17165869Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171669478Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"[2]R flush_numbers[0..12]: [326, 327, 326, 330, 328, 329, 329, 326, 330, 330, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171680097Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"[2]R generation[0..12]: [178, 180, 178, 182, 180, 181, 181, 178, 182, 182, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171690495Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171701324Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171710632Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171720181Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Next flush: 334","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.17172935Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171738548Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"":"downstairs","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171749467Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171758125Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"client":"0","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171767624Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"client":"1","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171777133Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"client":"2","":"downstairs","session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171786821Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.17179574Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"ae423c0a-6ca3-4cbe-9c60-d8c7daa6e0aa is now active with session: ba2f7627-8259-4b99-a2fd-74ebd82dde65","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171805369Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN"Wait for a query_work_queue command to finish before sending IO ,"pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171819536Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171827065Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873,"session_id":"ba2f7627-8259-4b99-a2fd-74ebd82dde65"} {"msg":"Activated sub_volume ae423c0a-6ca3-4cbe-9c60-d8c7daa6e0aa","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:16:57.171835944Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 99 len 20480 data: 19 18 19 15 16 02/30 Write block 72 len 24576 data: 17 13 13 13 14 13 03/30 Read block 486 len 36864 04/30 Read block 11 len 40960 05/30 Write block 357 len 16384 data: 16 14 10 12 06/30 Write block 181 len 16384 data: 21 21 21 17 07/30 Write block 546 len 40960 data: 11 10 12 13 12 13 12 13 14 16 08/30 Flush 09/30 Write block 124 len 36864 data: 14 13 14 15 13 13 15 16 16 10/30 Read block 279 len 8192 11/30 Write block 103 len 24576 data: 17 12 12 13 14 13 12/30 Write block 409 len 28672 data: 9 11 12 11 13 12 11 13/30 Read block 384 len 36864 14/30 Write block 171 len 36864 data: 16 15 15 15 15 18 20 21 19 15/30 Read block 389 len 12288 16/30 Flush 17/30 Write block 412 len 40960 data: 12 14 13 12 11 11 12 10 9 10 18/30 Read block 365 len 16384 19/30 Read block 432 len 28672 20/30 Write block 120 len 40960 data: 13 13 12 12 15 14 15 16 14 14 21/30 Read block 333 len 36864 22/30 Read block 153 len 24576 23/30 Flush 24/30 Write block 286 len 20480 data: 14 11 13 13 13 25/30 Read block 244 len 4096 26/30 Write block 123 len 4096 data: 13 27/30 Read block 165 len 36864 28/30 Write block 88 len 40960 data: 18 18 16 12 10 9 11 14 16 19 29/30 Write block 136 len 32768 data: 10 11 13 18 22 19 17 17 30/30 Write block 536 len 32768 data: 17 17 16 16 14 15 16 13 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Sent Done false 1001 Acked Write 6 Done Sent Done false 1002 Acked Read 9 Done Sent Done false 1003 Acked Read 10 Done Sent Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Write 4 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Read 9 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Write 10 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 5 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 2 34 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:16:57.181522998Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1873} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 326 326 326 F F F 1 020-039 180 180 180 327 327 327 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 184 182 184 334 330 334 F F F <--- 4 080-099 184 180 184 334 328 334 T F T <--- 5 100-119 184 181 184 335 329 335 F F F <--- 6 120-139 184 181 184 336 329 336 T F T <--- 7 140-159 178 178 178 326 326 326 T F T <--- 8 160-179 184 182 184 335 330 335 F F F <--- 9 180-199 184 182 184 334 330 334 F F F <--- 10 200-219 175 175 175 323 323 323 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 178 178 178 326 326 326 F F F 13 260-279 182 182 182 331 331 331 F F F 14 280-299 182 182 182 331 331 331 T F T <--- 15 300-319 180 180 180 327 327 327 F F F 16 320-339 183 183 183 333 333 333 F F F 17 340-359 184 180 184 334 327 334 F F F <--- 18 360-379 184 178 184 334 326 334 F F F <--- 19 380-399 181 181 181 329 329 329 F F F 20 400-419 184 175 184 336 323 336 F F F <--- 21 420-439 184 177 184 336 325 336 F F F <--- 22 440-459 178 178 178 326 326 326 F F F 23 460-479 178 178 178 326 326 326 F F F 24 480-499 182 182 182 331 331 331 F F F 25 500-519 182 182 182 331 331 331 F F F 26 520-539 181 181 181 329 329 329 T F T <--- 27 540-559 184 182 184 334 332 334 T F T <--- 28 560-579 183 183 183 333 333 333 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 184, Max flush: 336 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 326 326 326 F F F 1 020-039 180 180 180 327 327 327 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 184 184 184 334 334 334 F F F 4 080-099 185 185 185 337 337 337 F F F 5 100-119 184 184 184 335 335 335 F F F 6 120-139 185 185 185 337 337 337 F F F 7 140-159 185 185 185 337 337 337 F F F 8 160-179 184 184 184 335 335 335 F F F 9 180-199 184 184 184 334 334 334 F F F 10 200-219 175 175 175 323 323 323 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 178 178 178 326 326 326 F F F 13 260-279 182 182 182 331 331 331 F F F 14 280-299 185 185 185 337 337 337 F F F 15 300-319 180 180 180 327 327 327 F F F 16 320-339 183 183 183 333 333 333 F F F 17 340-359 184 184 184 334 334 334 F F F 18 360-379 184 184 184 334 334 334 F F F 19 380-399 181 181 181 329 329 329 F F F 20 400-419 184 184 184 336 336 336 F F F 21 420-439 184 184 184 336 336 336 F F F 22 440-459 178 178 178 326 326 326 F F F 23 460-479 178 178 178 326 326 326 F F F 24 480-499 182 182 182 331 331 331 F F F 25 500-519 182 182 182 331 331 331 F F F 26 520-539 185 185 185 337 337 337 F F F 27 540-559 185 185 185 337 337 337 F F F 28 560-579 183 183 183 333 333 333 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 185, Max flush: 337 Begin loop 93 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.438449037Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:01.445690799Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.445747631Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.452826826Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453088538Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453113774Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453124762Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453136321Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881} {"msg":"Crucible 013c1c68-eb6e-49c9-83de-639646e981b4 has session id: e25f3e57-8fce-4b20-a296-261a362cf602","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453489099Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Upstairs opts: Upstairs UUID: 013c1c68-eb6e-49c9-83de-639646e981b4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453518195Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Crucible stats registered with UUID: 013c1c68-eb6e-49c9-83de-639646e981b4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453529433Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:01.453538532Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.45369193Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"io task","client":"0","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453728264Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"io task","client":"1","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453739483Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"io task","client":"2","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453820471Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"task":"crutest"} {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453838348Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881} {"msg":"013c1c68-eb6e-49c9-83de-639646e981b4 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453862105Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453871733Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"client":"0","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453881532Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"client":"1","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.45389119Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"client":"2","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453902579Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"io task","client":"2","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453912587Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"io task","client":"0","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.453922726Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"io task","client":"1","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"ds_connection connected from Ok(127.0.0.1:42470)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.454011803Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"io task","client":"1","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"ds_connection connected from Ok(127.0.0.1:40647)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.454038349Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"io task","client":"0","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"ds_connection connected from Ok(127.0.0.1:45277)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.454049877Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"io task","client":"2","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.454703912Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"client":"2","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.454746335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"client":"1","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.454768562Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"client":"0","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455026474Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455035913Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"[0]R flush_numbers[0..12]: [326, 327, 326, 334, 337, 335, 337, 337, 335, 334, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455061089Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"[0]R generation[0..12]: [178, 180, 178, 184, 185, 184, 185, 185, 184, 184, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455072348Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455083346Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"[1]R flush_numbers[0..12]: [326, 327, 326, 334, 337, 335, 337, 337, 335, 334, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455095834Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"[1]R generation[0..12]: [178, 180, 178, 184, 185, 184, 185, 185, 184, 184, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455106473Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455117501Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"[2]R flush_numbers[0..12]: [326, 327, 326, 334, 337, 335, 337, 337, 335, 334, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.45512824Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"[2]R generation[0..12]: [178, 180, 178, 184, 185, 184, 185, 185, 184, 184, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455138988Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455149916Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Max found gen is 186","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455159245Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Generation requested: 186 >= found:186","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455169094Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Next flush: 338","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455178372Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455187721Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"":"downstairs","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455197409Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455206348Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"client":"0","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455215967Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"client":"1","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455225645Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"client":"2","":"downstairs","session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455235494Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Wait for a query_work_queue command to finish before sending IO Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455250232Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"013c1c68-eb6e-49c9-83de-639646e981b4 is now active with session: e25f3e57-8fce-4b20-a296-261a362cf602","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.45526023Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455269409Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455276958Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881,"session_id":"e25f3e57-8fce-4b20-a296-261a362cf602"} {"msg":"Activated sub_volume 013c1c68-eb6e-49c9-83de-639646e981b4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:01.455286017Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 334 len 32768 data: 13 11 12 9 10 11 11 14 02/30 Flush 03/30 Write block 193 len 16384 data: 17 18 18 16 04/30 Write block 254 len 36864 data: 7 8 7 6 6 9 7 13 16 05/30 Read block 369 len 28672 06/30 Read block 186 len 32768 07/30 Write block 168 len 32768 data: 17 16 15 17 16 16 16 16 08/30 Read block 466 len 28672 09/30 Write block 496 len 40960 data: 11 13 12 14 12 11 10 8 8 13 10/30 Read block 17 len 36864 11/30 Read block 180 len 12288 12/30 Read block 195 len 32768 13/30 Write block 424 len 36864 data: 13 14 15 17 14 13 12 15 18 14/30 Write block 518 len 40960 data: 14 13 13 15 15 11 11 11 12 14 15/30 Read block 516 len 12288 16/30 Write block 97 len 32768 data: 20 19 20 19 20 16 18 13 17/30 Read block 233 len 32768 18/30 Read block 441 len 40960 19/30 Write block 152 len 20480 data: 11 9 8 6 6 20/30 Read block 44 len 20480 21/30 Read block 387 len 36864 22/30 Write block 381 len 8192 data: 9 9 23/30 Read block 94 len 12288 24/30 Read block 96 len 16384 25/30 Read block 107 len 16384 26/30 Write block 94 len 4096 data: 12 27/30 Write block 528 len 40960 data: 14 13 12 14 17 21 19 20 18 18 28/30 Write block 266 len 24576 data: 21 20 18 19 16 18 29/30 Read block 101 len 20480 30/30 Write block 371 len 24576 data: 12 13 13 15 15 17 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 4 Sent Done Done false 1003 Acked Write 9 Sent Done Done false 1004 Acked Read 7 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Read 10 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:01.465435433Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1881} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 326 326 326 F F F 1 020-039 180 180 180 327 327 327 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 184 184 184 334 334 334 F F F 4 080-099 185 185 185 337 337 337 F T T <--- 5 100-119 184 184 184 335 335 335 F T T <--- 6 120-139 185 185 185 337 337 337 F F F 7 140-159 185 185 185 337 337 337 F T T <--- 8 160-179 184 184 184 335 335 335 F T T <--- 9 180-199 184 184 184 334 334 334 F T T <--- 10 200-219 175 175 175 323 323 323 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 178 178 178 326 326 326 F T T <--- 13 260-279 182 182 182 331 331 331 F T T <--- 14 280-299 185 185 185 337 337 337 F F F 15 300-319 180 180 180 327 327 327 F F F 16 320-339 183 186 186 333 338 338 F F F <--- 17 340-359 184 186 186 334 338 338 F F F <--- 18 360-379 184 184 184 334 334 334 F T T <--- 19 380-399 181 181 181 329 329 329 F T T <--- 20 400-419 184 184 184 336 336 336 F F F 21 420-439 184 184 184 336 336 336 F T T <--- 22 440-459 178 178 178 326 326 326 F F F 23 460-479 178 178 178 326 326 326 F F F 24 480-499 182 182 182 331 331 331 F T T <--- 25 500-519 182 182 182 331 331 331 F T T <--- 26 520-539 185 185 185 337 337 337 F T T <--- 27 540-559 185 185 185 337 337 337 F F F 28 560-579 183 183 183 333 333 333 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 186, Max flush: 338 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 178 178 178 326 326 326 F F F 1 020-039 180 180 180 327 327 327 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 184 184 184 334 334 334 F F F 4 080-099 187 187 187 339 339 339 F F F 5 100-119 187 187 187 339 339 339 F F F 6 120-139 185 185 185 337 337 337 F F F 7 140-159 187 187 187 339 339 339 F F F 8 160-179 187 187 187 339 339 339 F F F 9 180-199 187 187 187 339 339 339 F F F 10 200-219 175 175 175 323 323 323 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 187 187 187 339 339 339 F F F 13 260-279 187 187 187 339 339 339 F F F 14 280-299 185 185 185 337 337 337 F F F 15 300-319 180 180 180 327 327 327 F F F 16 320-339 186 186 186 338 338 338 F F F 17 340-359 186 186 186 338 338 338 F F F 18 360-379 187 187 187 339 339 339 F F F 19 380-399 187 187 187 339 339 339 F F F 20 400-419 184 184 184 336 336 336 F F F 21 420-439 187 187 187 339 339 339 F F F 22 440-459 178 178 178 326 326 326 F F F 23 460-479 178 178 178 326 326 326 F F F 24 480-499 187 187 187 339 339 339 F F F 25 500-519 187 187 187 339 339 339 F F F 26 520-539 187 187 187 339 339 339 F F F 27 540-559 185 185 185 337 337 337 F F F 28 560-579 183 183 183 333 333 333 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 187, Max flush: 339 Begin loop 94 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.732951355Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.741304804Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.741594351Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.741620267Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.741629006Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.741640544Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889} {"msg":"Crucible 051be66a-9430-4572-80f5-d25459ac4a42 has session id: ed1a072f-10b9-4294-9c0c-f683ff2c5768","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.741994733Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Upstairs opts: Upstairs UUID: 051be66a-9430-4572-80f5-d25459ac4a42, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742033247Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Crucible stats registered with UUID: 051be66a-9430-4572-80f5-d25459ac4a42","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742044385Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:05.742055934Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742193074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"io task","client":"2","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.7422169Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"io task","client":"0","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742228529Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"io task","client":"1","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742330674Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"task":"crutest"} {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742347941Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889} {"msg":"051be66a-9430-4572-80f5-d25459ac4a42 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742371338Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742380546Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"client":"0","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742390255Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"client":"1","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742399834Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"client":"2","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742409252Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"io task","client":"2","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742419281Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"io task","client":"0","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742429119Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"io task","client":"1","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"ds_connection connected from Ok(127.0.0.1:46823)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742471173Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"io task","client":"2","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"ds_connection connected from Ok(127.0.0.1:59762)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.742482021Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"io task","client":"0","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"ds_connection connected from Ok(127.0.0.1:49040)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.74249227Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"io task","client":"1","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743184929Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"client":"0","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743219084Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"client":"1","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743260318Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"client":"2","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743484415Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743504052Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"[0]R flush_numbers[0..12]: [326, 327, 326, 334, 339, 339, 337, 339, 339, 339, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743550485Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"[0]R generation[0..12]: [178, 180, 178, 184, 187, 187, 185, 187, 187, 187, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743561754Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743572602Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"[1]R flush_numbers[0..12]: [326, 327, 326, 334, 339, 339, 337, 339, 339, 339, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743583291Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"[1]R generation[0..12]: [178, 180, 178, 184, 187, 187, 185, 187, 187, 187, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743593749Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743604577Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"[2]R flush_numbers[0..12]: [326, 327, 326, 334, 339, 339, 337, 339, 339, 339, 323, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743614996Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"[2]R generation[0..12]: [178, 180, 178, 184, 187, 187, 185, 187, 187, 187, 175, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743625584Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743636453Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Max found gen is 188","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743648091Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.74365798Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Next flush: 340","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743667528Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743677037Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"":"downstairs","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743686665Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743695304Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"client":"0","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743704843Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"client":"1","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743714411Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"client":"2","":"downstairs","session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.74372407Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743732989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Wait for a query_work_queue command to finish before sending IO 051be66a-9430-4572-80f5-d25459ac4a42 is now active with session: ed1a072f-10b9-4294-9c0c-f683ff2c5768","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743747007Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743756265Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743763844Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889,"session_id":"ed1a072f-10b9-4294-9c0c-f683ff2c5768"} {"msg":"Activated sub_volume 051be66a-9430-4572-80f5-d25459ac4a42","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:05.743772793Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 244 len 20480 data: 18 17 18 16 14 02/30 Read block 595 len 16384 03/30 Read block 103 len 4096 04/30 Write block 371 len 28672 data: 13 14 14 16 16 18 18 05/30 Write block 14 len 28672 data: 17 16 15 16 14 12 12 06/30 Write block 182 len 24576 data: 22 22 18 14 15 12 07/30 Write block 312 len 16384 data: 19 18 15 16 08/30 Read block 215 len 4096 09/30 Read block 484 len 40960 10/30 Write block 135 len 4096 data: 12 11/30 Flush 12/30 Flush 13/30 Write block 61 len 20480 data: 13 14 12 16 17 14/30 Write block 558 len 28672 data: 20 19 19 19 17 16 17 15/30 Write block 257 len 36864 data: 7 7 10 8 14 17 18 19 20 16/30 Read block 518 len 24576 17/30 Read block 533 len 28672 18/30 Write block 308 len 28672 data: 14 16 16 19 20 19 16 19/30 Read block 323 len 8192 20/30 Read block 512 len 36864 21/30 Read block 112 len 24576 22/30 Write block 202 len 24576 data: 12 11 15 13 12 10 23/30 Read block 259 len 24576 24/30 Read block 82 len 16384 25/30 Write block 458 len 24576 data: 23 22 20 17 13 12 26/30 Read block 488 len 32768 27/30 Write block 473 len 32768 data: 20 19 16 11 9 6 6 6 28/30 Write block 242 len 20480 data: 17 17 19 18 19 29/30 Write block 296 len 36864 data: 9 9 11 9 7 8 8 10 12 30/30 Write block 441 len 24576 data: 18 20 17 17 18 17 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Sent false 1001 Acked Read 4 Done Done Sent false 1002 Acked Read 1 Done Done Sent false 1003 Acked Write 7 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Write 4 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Write 9 Done Done Sent false 1015 Acked Read 6 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Write 7 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Read 9 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Read 8 Done Done Sent false 1026 Acked Write 8 Sent Sent Sent false 1027 Acked Write 5 Sent Sent Sent false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 4 4 30 38 Done 26 26 0 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1011)) Some(JobId(1011)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:05.752286728Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1889} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 178 340 340 326 F F F <--- 1 020-039 188 188 180 340 340 327 F F F <--- 2 040-059 178 178 178 326 326 326 F F F 3 060-079 184 184 184 334 334 334 T T F <--- 4 080-099 187 187 187 339 339 339 F F F 5 100-119 187 187 187 339 339 339 F F F 6 120-139 188 188 185 340 340 337 F F F <--- 7 140-159 187 187 187 339 339 339 F F F 8 160-179 187 187 187 339 339 339 F F F 9 180-199 188 188 187 340 340 339 F F F <--- 10 200-219 175 175 175 323 323 323 T T F <--- 11 220-239 178 178 178 326 326 326 F F F 12 240-259 188 188 187 340 340 339 T T F <--- 13 260-279 187 187 187 339 339 339 T T F <--- 14 280-299 185 185 185 337 337 337 T T F <--- 15 300-319 188 188 180 340 340 327 T T F <--- 16 320-339 186 186 186 338 338 338 F F F 17 340-359 186 186 186 338 338 338 F F F 18 360-379 188 188 187 340 340 339 F F F <--- 19 380-399 187 187 187 339 339 339 F F F 20 400-419 184 184 184 336 336 336 F F F 21 420-439 187 187 187 339 339 339 F F F 22 440-459 178 178 178 326 326 326 T T F <--- 23 460-479 178 178 178 326 326 326 T T F <--- 24 480-499 187 187 187 339 339 339 T T F <--- 25 500-519 187 187 187 339 339 339 F F F 26 520-539 187 187 187 339 339 339 F F F 27 540-559 185 185 185 337 337 337 T T F <--- 28 560-579 183 183 183 333 333 333 T T F <--- 29 580-599 175 175 175 323 323 323 F F F Max gen: 188, Max flush: 340 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 340 340 340 F F F 1 020-039 188 188 188 340 340 340 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 189 189 189 341 341 341 F F F 4 080-099 187 187 187 339 339 339 F F F 5 100-119 187 187 187 339 339 339 F F F 6 120-139 188 188 188 340 340 340 F F F 7 140-159 187 187 187 339 339 339 F F F 8 160-179 187 187 187 339 339 339 F F F 9 180-199 188 188 188 340 340 340 F F F 10 200-219 189 189 189 341 341 341 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 189 189 189 341 341 341 F F F 13 260-279 189 189 189 341 341 341 F F F 14 280-299 189 189 189 341 341 341 F F F 15 300-319 189 189 189 341 341 341 F F F 16 320-339 186 186 186 338 338 338 F F F 17 340-359 186 186 186 338 338 338 F F F 18 360-379 188 188 188 340 340 340 F F F 19 380-399 187 187 187 339 339 339 F F F 20 400-419 184 184 184 336 336 336 F F F 21 420-439 187 187 187 339 339 339 F F F 22 440-459 189 189 189 341 341 341 F F F 23 460-479 189 189 189 341 341 341 F F F 24 480-499 189 189 189 341 341 341 F F F 25 500-519 187 187 187 339 339 339 F F F 26 520-539 187 187 187 339 339 339 F F F 27 540-559 189 189 189 341 341 341 F F F 28 560-579 189 189 189 341 341 341 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 189, Max flush: 341 Begin loop 95 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.017780826Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:10.025082408Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.025156328Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.032281166Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.032611448Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.032671649Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.032680488Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.032696226Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897} {"msg":"Crucible 31a1440f-49fa-4e74-a522-6056bbf0fa9a has session id: c78dbd6a-691e-42bf-8253-ecc5d7552ea9","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.032997571Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Upstairs opts: Upstairs UUID: 31a1440f-49fa-4e74-a522-6056bbf0fa9a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033041075Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Crucible stats registered with UUID: 31a1440f-49fa-4e74-a522-6056bbf0fa9a","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033051864Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:10.033060952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.03321274Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"io task","client":"2","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033235217Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"io task","client":"0","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033245215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"io task","client":"1","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033370117Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"task":"crutest"} {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033389554Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897} {"msg":"31a1440f-49fa-4e74-a522-6056bbf0fa9a active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033428538Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033437707Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"client":"0","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033447286Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"client":"1","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033456824Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"client":"2","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033466333Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"io task","client":"2","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033476232Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"io task","client":"0","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.03348597Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"io task","client":"1","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"ds_connection connected from Ok(127.0.0.1:64648)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033587535Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"io task","client":"2","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"ds_connection connected from Ok(127.0.0.1:64720)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.033613551Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"io task","client":"0","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"ds_connection connected from Ok(127.0.0.1:52191)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.03362491Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"io task","client":"1","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034317469Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"client":"1","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034335416Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"client":"0","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034350074Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"client":"2","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034628083Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034640491Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"[0]R flush_numbers[0..12]: [340, 340, 326, 341, 339, 339, 340, 339, 339, 340, 341, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034689754Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"[0]R generation[0..12]: [188, 188, 178, 189, 187, 187, 188, 187, 187, 188, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034701112Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034711901Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"[1]R flush_numbers[0..12]: [340, 340, 326, 341, 339, 339, 340, 339, 339, 340, 341, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034722419Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"[1]R generation[0..12]: [188, 188, 178, 189, 187, 187, 188, 187, 187, 188, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034732808Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034743616Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"[2]R flush_numbers[0..12]: [340, 340, 326, 341, 339, 339, 340, 339, 339, 340, 341, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034754135Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"[2]R generation[0..12]: [188, 188, 178, 189, 187, 187, 188, 187, 187, 188, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034764383Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034775102Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.03478443Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034794089Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Next flush: 342","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034803238Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034812576Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"":"downstairs","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034822195Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034832063Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"client":"0","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034841522Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"client":"1","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034850971Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"client":"2","":"downstairs","session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034860609Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034869588Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"31a1440f-49fa-4e74-a522-6056bbf0fa9a is now active with session: c78dbd6a-691e-42bf-8253-ecc5d7552ea9","v":0Wait for a query_work_queue command to finish before sending IO ,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034884006Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034893334Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034900853Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897,"session_id":"c78dbd6a-691e-42bf-8253-ecc5d7552ea9"} {"msg":"Activated sub_volume 31a1440f-49fa-4e74-a522-6056bbf0fa9a","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:10.034909792Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 552 len 20480 data: 13 14 15 17 18 02/30 Read block 180 len 24576 03/30 Read block 1 len 28672 04/30 Write block 178 len 16384 data: 22 20 23 22 05/30 Read block 449 len 24576 06/30 Read block 545 len 24576 07/30 Write block 63 len 40960 data: 13 17 18 15 15 14 15 15 17 18 08/30 Read block 454 len 20480 09/30 Read block 123 len 36864 10/30 Write block 439 len 24576 data: 19 17 19 21 18 18 11/30 Write block 458 len 8192 data: 24 23 12/30 Write block 179 len 8192 data: 21 24 13/30 Write block 163 len 20480 data: 20 22 22 20 18 14/30 Write block 23 len 24576 data: 10 10 14 14 12 14 15/30 Write block 408 len 24576 data: 10 10 12 13 13 15 16/30 Write block 571 len 28672 data: 15 16 13 16 17 20 18 17/30 Read block 434 len 20480 18/30 Write block 374 len 32768 data: 17 17 19 19 14 16 14 10 19/30 Write block 160 len 40960 data: 13 11 12 21 23 23 21 19 18 17 20/30 Read block 364 len 32768 21/30 Read block 157 len 4096 22/30 Write block 100 len 20480 data: 20 21 17 19 14 23/30 Flush 24/30 Write block 292 len 4096 data: 15 25/30 Read block 34 len 36864 26/30 Read block 195 len 20480 27/30 Write block 420 len 12288 data: 10 11 15 28/30 Read block 330 len 16384 29/30 Write block 471 len 12288 data: 18 18 21 30/30 Write block 383 len 12288 data: 9 10 7 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Sent Done Done false 1001 Acked Read 6 Sent Done Done false 1002 Acked Read 7 Sent Done Done false 1003 Acked Write 4 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Write 10 Sent Done Done false 1007 Acked Read 5 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Write 6 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Read 8 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:10.043697188Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1897} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 340 340 340 F F F 1 020-039 188 190 190 340 342 342 F F F <--- 2 040-059 178 178 178 326 326 326 F F F 3 060-079 189 190 190 341 342 342 F F F <--- 4 080-099 187 187 187 339 339 339 F F F 5 100-119 187 190 190 339 342 342 F F F <--- 6 120-139 188 188 188 340 340 340 F F F 7 140-159 187 187 187 339 339 339 F F F 8 160-179 187 190 190 339 342 342 F F F <--- 9 180-199 188 190 190 340 342 342 F F F <--- 10 200-219 189 189 189 341 341 341 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 189 189 189 341 341 341 F F F 13 260-279 189 189 189 341 341 341 F F F 14 280-299 189 189 189 341 341 341 F T T <--- 15 300-319 189 189 189 341 341 341 F F F 16 320-339 186 186 186 338 338 338 F F F 17 340-359 186 186 186 338 338 338 F F F 18 360-379 188 190 190 340 342 342 F F F <--- 19 380-399 187 190 190 339 342 342 F T T <--- 20 400-419 184 190 190 336 342 342 F F F <--- 21 420-439 187 190 190 339 342 342 F T T <--- 22 440-459 189 190 190 341 342 342 F F F <--- 23 460-479 189 189 189 341 341 341 F T T <--- 24 480-499 189 189 189 341 341 341 F F F 25 500-519 187 187 187 339 339 339 F F F 26 520-539 187 187 187 339 339 339 F F F 27 540-559 189 190 190 341 342 342 F F F <--- 28 560-579 189 190 190 341 342 342 F F F <--- 29 580-599 175 175 175 323 323 323 F F F Max gen: 190, Max flush: 342 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 95 Verifying data now No change after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 340 340 340 F F F 1 020-039 190 190 190 342 342 342 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 190 190 190 342 342 342 F F F 4 080-099 187 187 187 339 339 339 F F F 5 100-119 190 190 190 342 342 342 F F F 6 120-139 188 188 188 340 340 340 F F F 7 140-159 187 187 187 339 339 339 F F F 8 160-179 190 190 190 342 342 342 F F F 9 180-199 190 190 190 342 342 342 F F F 10 200-219 189 189 189 341 341 341 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 189 189 189 341 341 341 F F F 13 260-279 189 189 189 341 341 341 F F F 14 280-299 191 191 191 343 343 343 F F F 15 300-319 189 189 189 341 341 341 F F F 16 320-339 186 186 186 338 338 338 F F F 17 340-359 186 186 186 338 338 338 F F F 18 360-379 190 190 190 342 342 342 F F F 19 380-399 191 191 191 343 343 343 F F F 20 400-419 190 190 190 342 342 342 F F F 21 420-439 191 191 191 343 343 343 F F F 22 440-459 190 190 190 342 342 342 F F F 23 460-479 191 191 191 343 343 343 F F F 24 480-499 189 189 189 341 341 341 F F F 25 500-519 187 187 187 339 339 339 F F F 26 520-539 187 187 187 339 339 339 F F F 27 540-559 190 190 190 342 342 342 F F F 28 560-579 190 190 190 342 342 342 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 191, Max flush: 343 Begin loop 96 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.304958032Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.313189039Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.313472098Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.313502683Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.313511182Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.313522751Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905} {"msg":"Crucible 2e873a70-aa81-4c0b-af6e-0018b0ac487e has session id: 5addb278-d640-4d45-bfe6-ec7b383733c4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.31386917Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Upstairs opts: Upstairs UUID: 2e873a70-aa81-4c0b-af6e-0018b0ac487e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.313892217Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Crucible stats registered with UUID: 2e873a70-aa81-4c0b-af6e-0018b0ac487e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.313902075Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:14.313911004Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314090987Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"io task","client":"0","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314128242Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"io task","client":"1","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.31413925Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"io task","client":"2","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314305136Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"task":"crutest"} {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314323983Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905} {"msg":"2e873a70-aa81-4c0b-af6e-0018b0ac487e active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314341121Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.31434999Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"client":"0","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314359558Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"client":"1","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314368997Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"client":"2","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314378835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"io task","client":"2","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314388674Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"io task","client":"0","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314400342Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"io task","client":"1","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"ds_connection connected from Ok(127.0.0.1:51137)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314471322Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"io task","client":"2","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"ds_connection connected from Ok(127.0.0.1:58211)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.31448742Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"io task","client":"0","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"ds_connection connected from Ok(127.0.0.1:56431)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.314497758Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"io task","client":"1","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315143574Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"client":"1","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315174209Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"client":"0","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315229211Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"client":"2","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315482634Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315525468Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"[0]R flush_numbers[0..12]: [340, 342, 326, 342, 339, 342, 340, 339, 342, 342, 341, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315594288Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"[0]R generation[0..12]: [188, 190, 178, 190, 187, 190, 188, 187, 190, 190, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315605366Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315616295Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"[1]R flush_numbers[0..12]: [340, 342, 326, 342, 339, 342, 340, 339, 342, 342, 341, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315626703Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"[1]R generation[0..12]: [188, 190, 178, 190, 187, 190, 188, 187, 190, 190, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315637111Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.31564957Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"[2]R flush_numbers[0..12]: [340, 342, 326, 342, 339, 342, 340, 339, 342, 342, 341, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315660248Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"[2]R generation[0..12]: [188, 190, 178, 190, 187, 190, 188, 187, 190, 190, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315670697Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315681445Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315690814Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315700502Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Next flush: 344","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315709731Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.31571889Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"":"downstairs","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315728578Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315737297Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"client":"0","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315747215Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"client":"1","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":Wait for a query_work_queue command to finish before sending IO 30,"time":"2026-04-15T18:17:14.315761503Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"client":"2","":"downstairs","session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315771432Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315780281Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"2e873a70-aa81-4c0b-af6e-0018b0ac487e is now active with session: 5addb278-d640-4d45-bfe6-ec7b383733c4","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315790029Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315800428Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315807897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905,"session_id":"5addb278-d640-4d45-bfe6-ec7b383733c4"} {"msg":"Activated sub_volume 2e873a70-aa81-4c0b-af6e-0018b0ac487e","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:14.315816755Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 396 len 4096 data: 13 03/30 Read block 238 len 24576 04/30 Write block 301 len 8192 data: 9 9 05/30 Write block 23 len 4096 data: 11 06/30 Read block 122 len 24576 07/30 Write block 560 len 36864 data: 20 20 18 17 18 14 11 12 10 08/30 Write block 386 len 20480 data: 8 10 13 13 12 09/30 Write block 333 len 36864 data: 13 14 12 13 10 11 12 12 15 10/30 Write block 404 len 8192 data: 14 15 11/30 Read block 187 len 12288 12/30 Write block 426 len 28672 data: 16 18 15 14 13 16 19 13/30 Read block 578 len 16384 14/30 Write block 1 len 40960 data: 6 9 9 11 11 11 11 10 12 13 15/30 Write block 14 len 16384 data: 18 17 16 17 16/30 Read block 104 len 32768 17/30 Read block 589 len 20480 18/30 Write block 427 len 36864 data: 19 16 15 14 17 20 20 23 24 19/30 Read block 467 len 24576 20/30 Write block 13 len 20480 data: 16 19 18 17 18 21/30 Write block 11 len 36864 data: 14 15 17 20 19 18 19 15 13 22/30 Write block 538 len 8192 data: 17 17 23/30 Read block 46 len 36864 24/30 Read block 29 len 28672 25/30 Read block 369 len 16384 26/30 Flush 27/30 Write block 540 len 28672 data: 15 16 17 14 11 10 12 28/30 Write block 29 len 24576 data: 13 13 10 10 10 9 29/30 Write block 171 len 40960 data: 18 17 17 17 17 19 21 23 22 25 30/30 Read block 536 len 8192 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 1 Done Done Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Write 2 Done Sent Done false 1004 Acked Write 1 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Write 9 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Read 3 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Write 4 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Read 5 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Write 9 Done Sent Done false 1021 Acked Write 2 Done Sent Done false 1022 Acked Read 9 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Write 10 Done Sent Done false 1029 Acked Read 2 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 1 29 Done 29 1 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1000)) Some(JobId(1025)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:14.324301675Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1905} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 188 192 345 340 345 F F F <--- 1 020-039 192 190 192 345 342 345 T F T <--- 2 040-059 178 178 178 326 326 326 F F F 3 060-079 190 190 190 342 342 342 F F F 4 080-099 187 187 187 339 339 339 F F F 5 100-119 190 190 190 342 342 342 F F F 6 120-139 188 188 188 340 340 340 F F F 7 140-159 187 187 187 339 339 339 F F F 8 160-179 190 190 190 342 342 342 T F T <--- 9 180-199 190 190 190 342 342 342 T F T <--- 10 200-219 189 189 189 341 341 341 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 189 189 189 341 341 341 F F F 13 260-279 189 189 189 341 341 341 F F F 14 280-299 191 191 191 343 343 343 F F F 15 300-319 192 189 192 345 341 345 F F F <--- 16 320-339 192 186 192 345 338 345 F F F <--- 17 340-359 192 186 192 345 338 345 F F F <--- 18 360-379 190 190 190 342 342 342 F F F 19 380-399 192 191 192 345 343 345 F T F <--- 20 400-419 192 190 192 345 342 345 F F F <--- 21 420-439 192 191 192 345 343 345 F F F <--- 22 440-459 190 190 190 342 342 342 F F F 23 460-479 191 191 191 343 343 343 F F F 24 480-499 189 189 189 341 341 341 F F F 25 500-519 187 187 187 339 339 339 F F F 26 520-539 192 187 192 345 339 345 F F F <--- 27 540-559 190 190 190 342 342 342 T F T <--- 28 560-579 192 190 192 345 342 345 F F F <--- 29 580-599 175 175 175 323 323 323 F F F Max gen: 192, Max flush: 345 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 345 345 345 F F F 1 020-039 193 193 193 346 346 346 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 190 190 190 342 342 342 F F F 4 080-099 187 187 187 339 339 339 F F F 5 100-119 190 190 190 342 342 342 F F F 6 120-139 188 188 188 340 340 340 F F F 7 140-159 187 187 187 339 339 339 F F F 8 160-179 193 193 193 346 346 346 F F F 9 180-199 193 193 193 346 346 346 F F F 10 200-219 189 189 189 341 341 341 F F F 11 220-239 178 178 178 326 326 326 F F F 12 240-259 189 189 189 341 341 341 F F F 13 260-279 189 189 189 341 341 341 F F F 14 280-299 191 191 191 343 343 343 F F F 15 300-319 192 192 192 345 345 345 F F F 16 320-339 192 192 192 345 345 345 F F F 17 340-359 192 192 192 345 345 345 F F F 18 360-379 190 190 190 342 342 342 F F F 19 380-399 192 192 192 345 345 345 F F F 20 400-419 192 192 192 345 345 345 F F F 21 420-439 192 192 192 345 345 345 F F F 22 440-459 190 190 190 342 342 342 F F F 23 460-479 191 191 191 343 343 343 F F F 24 480-499 189 189 189 341 341 341 F F F 25 500-519 187 187 187 339 339 339 F F F 26 520-539 192 192 192 345 345 345 F F F 27 540-559 193 193 193 346 346 346 F F F 28 560-579 192 192 192 345 345 345 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 193, Max flush: 346 Begin loop 97 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.574044264Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.582190613Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.582492779Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.582518106Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.582526504Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.582537893Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913} {"msg":"Crucible 6a91e231-2cbc-4391-8199-6d86fd0e1736 has session id: 1466849a-40bb-4510-9e61-aee2d4a9295d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.582928486Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Upstairs opts: Upstairs UUID: 6a91e231-2cbc-4391-8199-6d86fd0e1736, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.582944933Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Crucible stats registered with UUID: 6a91e231-2cbc-4391-8199-6d86fd0e1736","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.582954772Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:18.58296385Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583164961Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"io task","client":"0","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583222323Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"io task","client":"2","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583235091Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"io task","client":"1","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583322268Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"task":"crutest"} {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583339376Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913} {"msg":"6a91e231-2cbc-4391-8199-6d86fd0e1736 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583373651Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583382989Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"client":"0","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583398337Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"client":"1","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583408025Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"client":"2","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583417574Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"io task","client":"2","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583427563Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"io task","client":"0","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583437421Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"io task","client":"1","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"ds_connection connected from Ok(127.0.0.1:38059)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583507191Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"io task","client":"2","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"ds_connection connected from Ok(127.0.0.1:46999)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583533127Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"io task","client":"0","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"ds_connection connected from Ok(127.0.0.1:56012)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.583544546Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"io task","client":"1","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584250842Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"client":"2","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584292156Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"client":"1","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584305464Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"client":"0","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584555068Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584596662Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"[0]R flush_numbers[0..12]: [345, 346, 326, 342, 339, 342, 340, 339, 346, 346, 341, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.58467519Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"[0]R generation[0..12]: [192, 193, 178, 190, 187, 190, 188, 187, 193, 193, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584686719Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584697817Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"[1]R flush_numbers[0..12]: [345, 346, 326, 342, 339, 342, 340, 339, 346, 346, 341, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584708495Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"[1]R generation[0..12]: [192, 193, 178, 190, 187, 190, 188, 187, 193, 193, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584718814Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584729612Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"[2]R flush_numbers[0..12]: [345, 346, 326, 342, 339, 342, 340, 339, 346, 346, 341, 326]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584740011Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"[2]R generation[0..12]: [192, 193, 178, 190, 187, 190, 188, 187, 193, 193, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584750499Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584761348Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Max found gen is 194","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584770636Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584780355Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Next flush: 347","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584791093Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584800442Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"":"downstairs","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584810071Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584818809Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"client":"0","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584828408Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"client":"1","":"downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584837886Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"client":"2","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584851335Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584860363Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"6a91e231-2cbc-4391-8199-6d86fd0e1736 is now active with session: 1466849a-40bb-4510-9e61-aee2d4a9295d","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584870222Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.58487914Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584886599Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913,"session_id":"1466849a-40bb-4510-9e61-aee2d4a9295d"} {"msg":"Activated sub_volume 6a91e231-2cbc-4391-8199-6d86fd0e1736","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:18.584895668Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 550 len 24576 data: 13 14 14 15 16 18 02/30 Read block 588 len 40960 03/30 Write block 236 len 12288 data: 10 12 14 04/30 Write block 126 len 4096 data: 16 05/30 Read block 261 len 28672 06/30 Write block 99 len 16384 data: 21 21 22 18 07/30 Read block 381 len 32768 08/30 Read block 29 len 16384 09/30 Flush 10/30 Write block 163 len 8192 data: 22 24 11/30 Read block 579 len 12288 12/30 Read block 359 len 16384 13/30 Read block 571 len 12288 14/30 Read block 44 len 36864 15/30 Read block 50 len 8192 16/30 Write block 502 len 12288 data: 11 9 9 17/30 Write block 445 len 12288 data: 19 18 15 18/30 Read block 285 len 40960 19/30 Write block 249 len 20480 data: 14 14 12 10 6 20/30 Read block 105 len 32768 21/30 Read block 295 len 12288 22/30 Read block 187 len 40960 23/30 Read block 383 len 28672 24/30 Write block 141 len 32768 data: 20 18 18 16 17 13 12 11 25/30 Write block 381 len 12288 data: 11 10 10 26/30 Read block 310 len 8192 27/30 Write block 416 len 20480 data: 12 12 13 11 11 28/30 Write block 316 len 40960 data: 19 20 19 20 17 11 9 8 8 9 29/30 Read block 170 len 24576 30/30 Write block 78 len 36864 data: 15 15 13 20 22 19 18 19 16 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Write 3 Done Sent Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Read 7 Done Sent Done false 1005 Acked Write 4 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Read 3 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Read 2 Done Sent Done false 1015 Acked Write 3 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 7 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Write 3 Done Sent Done false 1025 Acked Read 2 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 28 1 30 Done 29 2 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1008)) None Some(JobId(1008)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:18.593854929Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1913} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 345 345 345 F F F 1 020-039 193 193 193 346 346 346 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 190 190 190 342 342 342 T F T <--- 4 080-099 194 187 194 347 339 347 T F T <--- 5 100-119 194 190 194 347 342 347 F F F <--- 6 120-139 194 188 194 347 340 347 F F F <--- 7 140-159 187 187 187 339 339 339 T F T <--- 8 160-179 193 193 193 346 346 346 T F T <--- 9 180-199 193 193 193 346 346 346 F F F 10 200-219 189 189 189 341 341 341 F F F 11 220-239 194 178 194 347 326 347 F F F <--- 12 240-259 189 189 189 341 341 341 T F T <--- 13 260-279 189 189 189 341 341 341 F F F 14 280-299 191 191 191 343 343 343 F F F 15 300-319 192 192 192 345 345 345 T F T <--- 16 320-339 192 192 192 345 345 345 T F T <--- 17 340-359 192 192 192 345 345 345 F F F 18 360-379 190 190 190 342 342 342 F F F 19 380-399 192 192 192 345 345 345 T F T <--- 20 400-419 192 192 192 345 345 345 T F T <--- 21 420-439 192 192 192 345 345 345 T F T <--- 22 440-459 190 190 190 342 342 342 T F T <--- 23 460-479 191 191 191 343 343 343 F F F 24 480-499 189 189 189 341 341 341 F F F 25 500-519 187 187 187 339 339 339 T F T <--- 26 520-539 192 192 192 345 345 345 F F F 27 540-559 194 193 194 347 346 347 F T F <--- 28 560-579 192 192 192 345 345 345 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 194, Max flush: 347 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 345 345 345 F F F 1 020-039 193 193 193 346 346 346 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 195 195 195 348 348 348 F F F 4 080-099 195 195 195 348 348 348 F F F 5 100-119 194 194 194 347 347 347 F F F 6 120-139 194 194 194 347 347 347 F F F 7 140-159 195 195 195 348 348 348 F F F 8 160-179 195 195 195 348 348 348 F F F 9 180-199 193 193 193 346 346 346 F F F 10 200-219 189 189 189 341 341 341 F F F 11 220-239 194 194 194 347 347 347 F F F 12 240-259 195 195 195 348 348 348 F F F 13 260-279 189 189 189 341 341 341 F F F 14 280-299 191 191 191 343 343 343 F F F 15 300-319 195 195 195 348 348 348 F F F 16 320-339 195 195 195 348 348 348 F F F 17 340-359 192 192 192 345 345 345 F F F 18 360-379 190 190 190 342 342 342 F F F 19 380-399 195 195 195 348 348 348 F F F 20 400-419 195 195 195 348 348 348 F F F 21 420-439 195 195 195 348 348 348 F F F 22 440-459 195 195 195 348 348 348 F F F 23 460-479 191 191 191 343 343 343 F F F 24 480-499 189 189 189 341 341 341 F F F 25 500-519 195 195 195 348 348 348 F F F 26 520-539 192 192 192 345 345 345 F F F 27 540-559 194 194 194 347 347 347 F F F 28 560-579 192 192 192 345 345 345 F F F 29 580-599 175 175 175 323 323 323 F F F Max gen: 195, Max flush: 348 Begin loop 98 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.85979926Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.868110455Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.868432768Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.868467833Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.868479491Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.868491299Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921} {"msg":"Crucible c906d109-064d-433a-98d0-4dd6c4c270b5 has session id: 54f49254-56e1-40bb-a6aa-b73ec47799ca","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.868850427Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Upstairs opts: Upstairs UUID: c906d109-064d-433a-98d0-4dd6c4c270b5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.868878463Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Crucible stats registered with UUID: c906d109-064d-433a-98d0-4dd6c4c270b5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.868889551Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:22.86889897Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869078744Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"io task","client":"1","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.86910171Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"io task","client":"2","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869111969Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"io task","client":"0","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869217353Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"task":"crutest"} {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869244039Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921} {"msg":"c906d109-064d-433a-98d0-4dd6c4c270b5 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869267776Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869277035Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"client":"0","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869286673Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"client":"1","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869296282Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"client":"2","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.86930756Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"io task","client":"2","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869317409Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"io task","client":"0","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869342665Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"io task","client":"1","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"ds_connection connected from Ok(127.0.0.1:38821)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869428083Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"io task","client":"0","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"ds_connection connected from Ok(127.0.0.1:63013)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.86944311Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"io task","client":"1","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"ds_connection connected from Ok(127.0.0.1:34852)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.869453789Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"io task","client":"2","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.87019717Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"client":"1","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870222536Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"client":"0","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870234305Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"client":"2","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870527512Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.87053799Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"[0]R flush_numbers[0..12]: [345, 346, 326, 348, 348, 347, 347, 348, 348, 346, 341, 347]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870576495Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"[0]R generation[0..12]: [192, 193, 178, 195, 195, 194, 194, 195, 195, 193, 189, 194]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870587863Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870598911Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"[1]R flush_numbers[0..12]: [345, 346, 326, 348, 348, 347, 347, 348, 348, 346, 341, 347]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.87060958Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"[1]R generation[0..12]: [192, 193, 178, 195, 195, 194, 194, 195, 195, 193, 189, 194]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870622218Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870633096Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"[2]R flush_numbers[0..12]: [345, 346, 326, 348, 348, 347, 347, 348, 348, 346, 341, 347]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870643595Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"[2]R generation[0..12]: [192, 193, 178, 195, 195, 194, 194, 195, 195, 193, 189, 194]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870653913Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870664692Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.87067391Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Generation requested: 196 >= found:196","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870683419Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Next flush: 349","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870692618Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870701796Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"":"downstairs","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870711685Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870720444Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"client":"0","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870730302Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"client":"1","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.8707449Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"client":"2","":"downstairs","session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870756808Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870765847Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"c906d109-064d-433a-98d0-4dd6c4c270b5 is now active with session: 54f49254-56e1-40bb-a6aa-b73ec47799ca","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870776835Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870786084Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2026-04-15T18:17:22.870797952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921,"session_id":"54f49254-56e1-40bb-a6aa-b73ec47799ca"} {"msg":"Activated sub_volume c906d109-064d-433a-98d0-4dd6c4c270b5","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:22.870807071Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 418 len 32768 data: 14 12 12 12 16 15 14 15 02/30 Flush 03/30 Flush 04/30 Write block 242 len 32768 data: 18 18 20 19 20 17 15 15 05/30 Read block 566 len 12288 06/30 Read block 451 len 16384 07/30 Read block 474 len 16384 08/30 Flush 09/30 Write block 150 len 4096 data: 11 10/30 Read block 119 len 20480 11/30 Read block 181 len 32768 12/30 Read block 1 len 24576 13/30 Write block 430 len 40960 data: 15 18 21 21 24 25 24 23 24 20 14/30 Flush 15/30 Flush 16/30 Write block 586 len 28672 data: 11 14 14 10 12 12 13 17/30 Flush 18/30 Write block 556 len 20480 data: 19 20 21 20 21 19/30 Read block 385 len 36864 20/30 Write block 6 len 12288 data: 12 12 11 21/30 Write block 464 len 32768 data: 9 8 12 15 19 17 18 19 22/30 Write block 226 len 32768 data: 16 15 17 18 19 17 13 12 23/30 Read block 398 len 40960 24/30 Read block 442 len 32768 25/30 Read block 107 len 12288 26/30 Write block 313 len 8192 data: 20 17 27/30 Read block 333 len 32768 28/30 Read block 128 len 16384 29/30 Write block 405 len 16384 data: 16 15 12 11 30/30 Write block 394 len 40960 data: 13 13 14 12 15 14 14 13 15 13 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Sent false 1001 Acked Flush 0 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Read 3 Done Done Sent false 1005 Acked Read 4 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 7 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Write 8 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Read 3 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Write 4 Done Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 2 30 33 Done 29 28 0 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:22.880480107Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1921} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 345 345 345 T T F <--- 1 020-039 193 193 193 346 346 346 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 195 195 195 348 348 348 F F F 4 080-099 195 195 195 348 348 348 F F F 5 100-119 194 194 194 347 347 347 F F F 6 120-139 194 194 194 347 347 347 F F F 7 140-159 196 196 195 352 352 348 F F F <--- 8 160-179 195 195 195 348 348 348 F F F 9 180-199 193 193 193 346 346 346 F F F 10 200-219 189 189 189 341 341 341 F F F 11 220-239 194 194 194 347 347 347 T T F <--- 12 240-259 196 196 195 351 351 348 F F F <--- 13 260-279 189 189 189 341 341 341 F F F 14 280-299 191 191 191 343 343 343 F F F 15 300-319 195 195 195 348 348 348 T T F <--- 16 320-339 195 195 195 348 348 348 F F F 17 340-359 192 192 192 345 345 345 F F F 18 360-379 190 190 190 342 342 342 F F F 19 380-399 195 195 195 348 348 348 T T F <--- 20 400-419 196 196 195 349 349 348 T T F <--- 21 420-439 196 196 195 352 352 348 F F F <--- 22 440-459 195 195 195 348 348 348 F F F 23 460-479 191 191 191 343 343 343 T T F <--- 24 480-499 189 189 189 341 341 341 F F F 25 500-519 195 195 195 348 348 348 F F F 26 520-539 192 192 192 345 345 345 F F F 27 540-559 194 194 194 347 347 347 T T F <--- 28 560-579 192 192 192 345 345 345 T T F <--- 29 580-599 196 196 175 354 354 323 F F F <--- Max gen: 196, Max flush: 354 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 197 197 197 355 355 355 F F F 1 020-039 193 193 193 346 346 346 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 195 195 195 348 348 348 F F F 4 080-099 195 195 195 348 348 348 F F F 5 100-119 194 194 194 347 347 347 F F F 6 120-139 194 194 194 347 347 347 F F F 7 140-159 196 196 196 352 352 352 F F F 8 160-179 195 195 195 348 348 348 F F F 9 180-199 193 193 193 346 346 346 F F F 10 200-219 189 189 189 341 341 341 F F F 11 220-239 197 197 197 355 355 355 F F F 12 240-259 196 196 196 351 351 351 F F F 13 260-279 189 189 189 341 341 341 F F F 14 280-299 191 191 191 343 343 343 F F F 15 300-319 197 197 197 355 355 355 F F F 16 320-339 195 195 195 348 348 348 F F F 17 340-359 192 192 192 345 345 345 F F F 18 360-379 190 190 190 342 342 342 F F F 19 380-399 197 197 197 355 355 355 F F F 20 400-419 197 197 197 355 355 355 F F F 21 420-439 196 196 196 352 352 352 F F F 22 440-459 195 195 195 348 348 348 F F F 23 460-479 197 197 197 355 355 355 F F F 24 480-499 189 189 189 341 341 341 F F F 25 500-519 195 195 195 348 348 348 F F F 26 520-539 192 192 192 345 345 345 F F F 27 540-559 197 197 197 355 355 355 F F F 28 560-579 197 197 197 355 355 355 F F F 29 580-599 196 196 196 354 354 354 F F F Max gen: 197, Max flush: 355 Begin loop 99 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.133130941Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:27.140338228Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.140400109Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.147472095Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.147740166Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51ca5fedcda7bf0cf4b8f1540dbb8d79ece61b55\",\n git_commit_timestamp: \"2026-04-15T17:55:14.000000000Z\",\n git_branch: \"alan/cmon-polish\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.147767632Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.147776001Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.147787459Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929} {"msg":"Crucible ff415e51-426a-4ac1-aaad-f986ce94f6f2 has session id: 55c51f96-71df-4dba-9b60-d47f642f7aba","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148115091Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Upstairs opts: Upstairs UUID: ff415e51-426a-4ac1-aaad-f986ce94f6f2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148143877Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Crucible stats registered with UUID: ff415e51-426a-4ac1-aaad-f986ce94f6f2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148157245Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:27.148166494Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148303414Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"io task","client":"2","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.14832623Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"io task","client":"0","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148336549Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"io task","client":"1","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148460721Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"task":"crutest"} {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148477808Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929} {"msg":"ff415e51-426a-4ac1-aaad-f986ce94f6f2 active request set","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148500915Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148510384Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"client":"0","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148519952Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"client":"1","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148529681Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"client":"2","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148539219Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"io task","client":"2","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148549198Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"io task","client":"0","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148558946Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"io task","client":"1","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"ds_connection connected from Ok(127.0.0.1:64647)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148650973Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"io task","client":"1","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"ds_connection connected from Ok(127.0.0.1:60210)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148676679Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"io task","client":"0","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"ds_connection connected from Ok(127.0.0.1:43723)","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.148751208Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"io task","client":"2","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149331064Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"client":"2","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.14935292Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"client":"1","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149379297Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"client":"0","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149674133Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149683872Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"[0]R flush_numbers[0..12]: [355, 346, 326, 348, 348, 347, 347, 352, 348, 346, 341, 355]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149719267Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"[0]R generation[0..12]: [197, 193, 178, 195, 195, 194, 194, 196, 195, 193, 189, 197]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149730575Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149741504Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"[1]R flush_numbers[0..12]: [355, 346, 326, 348, 348, 347, 347, 352, 348, 346, 341, 355]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149752172Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"[1]R generation[0..12]: [197, 193, 178, 195, 195, 194, 194, 196, 195, 193, 189, 197]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149762591Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149773529Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"[2]R flush_numbers[0..12]: [355, 346, 326, 348, 348, 347, 347, 352, 348, 346, 341, 355]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149784057Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"[2]R generation[0..12]: [197, 193, 178, 195, 195, 194, 194, 196, 195, 193, 189, 197]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149794616Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149808014Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149817333Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Generation requested: 198 >= found:198","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149826951Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Next flush: 356","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.14983611Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149845418Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"":"downstairs","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149854997Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149863586Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"client":"0","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149873194Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"client":"1","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Transition from WaitQuorum to ActiveWait for a query_work_queue command to finish before sending IO ","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149887592Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"client":"2","":"downstairs","session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149897571Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.14990656Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"ff415e51-426a-4ac1-aaad-f986ce94f6f2 is now active with session: 55c51f96-71df-4dba-9b60-d47f642f7aba","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149916368Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149925497Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149933026Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929,"session_id":"55c51f96-71df-4dba-9b60-d47f642f7aba"} {"msg":"Activated sub_volume ff415e51-426a-4ac1-aaad-f986ce94f6f2","v":0,"name":"crucible","level":30,"time":"2026-04-15T18:17:27.149941954Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 194 len 40960 data: 19 19 17 16 17 16 14 14 13 12 02/30 Read block 431 len 32768 03/30 Read block 39 len 20480 04/30 Write block 263 len 40960 data: 19 20 21 22 21 19 20 17 19 17 05/30 Write block 579 len 28672 data: 16 15 14 13 15 14 14 06/30 Flush 07/30 Flush 08/30 Write block 233 len 32768 data: 13 10 9 11 13 15 14 13 09/30 Read block 387 len 20480 10/30 Flush 11/30 Write block 283 len 4096 data: 14 12/30 Write block 10 len 12288 data: 14 15 16 13/30 Read block 103 len 24576 14/30 Read block 503 len 40960 15/30 Flush 16/30 Write block 226 len 36864 data: 17 16 18 19 20 18 14 14 11 17/30 Flush 18/30 Write block 231 len 28672 data: 19 15 15 12 10 12 14 19/30 Write block 445 len 4096 data: 20 20/30 Read block 308 len 12288 21/30 Write block 146 len 12288 data: 14 13 12 22/30 Read block 206 len 40960 23/30 Read block 458 len 32768 24/30 Read block 25 len 24576 25/30 Read block 319 len 4096 26/30 Read block 549 len 20480 27/30 Write block 579 len 8192 data: 17 16 28/30 Read block 260 len 40960 29/30 Read block 499 len 36864 30/30 Read block 440 len 24576 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Sent Done Done false 1001 Acked Read 8 Sent Done Done false 1002 Acked Read 5 Sent Done Done false 1003 Acked Write 10 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 5 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Read 6 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Read 10 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Write 2 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Read 6 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 0 31 Done 0 29 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-04-15T18:17:27.160061645Z","hostname":"w-01KP95A0FYJFBCSZ7GRQJANDZN","pid":1929} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 197 198 198 355 359 359 F F F <--- 1 020-039 193 193 193 346 346 346 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 195 195 195 348 348 348 F F F 4 080-099 195 195 195 348 348 348 F F F 5 100-119 194 194 194 347 347 347 F F F 6 120-139 194 194 194 347 347 347 F F F 7 140-159 196 196 196 352 352 352 F T T <--- 8 160-179 195 195 195 348 348 348 F F F 9 180-199 193 198 198 346 356 356 F F F <--- 10 200-219 189 198 198 341 356 356 F F F <--- 11 220-239 197 198 198 355 360 360 F T T <--- 12 240-259 196 198 198 351 358 358 F F F <--- 13 260-279 189 198 198 341 356 356 F F F <--- 14 280-299 191 198 198 343 359 359 F F F <--- 15 300-319 197 197 197 355 355 355 F F F 16 320-339 195 195 195 348 348 348 F F F 17 340-359 192 192 192 345 345 345 F F F 18 360-379 190 190 190 342 342 342 F F F 19 380-399 197 197 197 355 355 355 F F F 20 400-419 197 197 197 355 355 355 F F F 21 420-439 196 196 196 352 352 352 F F F 22 440-459 195 195 195 348 348 348 F T T <--- 23 460-479 197 197 197 355 355 355 F F F 24 480-499 189 189 189 341 341 341 F F F 25 500-519 195 195 195 348 348 348 F F F 26 520-539 192 192 192 345 345 345 F F F 27 540-559 197 197 197 355 355 355 F F F 28 560-579 197 198 198 355 356 356 F T T <--- 29 580-599 196 198 198 354 356 356 F T T <--- Max gen: 198, Max flush: 360 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 198 198 198 359 359 359 F F F 1 020-039 193 193 193 346 346 346 F F F 2 040-059 178 178 178 326 326 326 F F F 3 060-079 195 195 195 348 348 348 F F F 4 080-099 195 195 195 348 348 348 F F F 5 100-119 194 194 194 347 347 347 F F F 6 120-139 194 194 194 347 347 347 F F F 7 140-159 199 199 199 361 361 361 F F F 8 160-179 195 195 195 348 348 348 F F F 9 180-199 198 198 198 356 356 356 F F F 10 200-219 198 198 198 356 356 356 F F F 11 220-239 199 199 199 361 361 361 F F F 12 240-259 198 198 198 358 358 358 F F F 13 260-279 198 198 198 356 356 356 F F F 14 280-299 198 198 198 359 359 359 F F F 15 300-319 197 197 197 355 355 355 F F F 16 320-339 195 195 195 348 348 348 F F F 17 340-359 192 192 192 345 345 345 F F F 18 360-379 190 190 190 342 342 342 F F F 19 380-399 197 197 197 355 355 355 F F F 20 400-419 197 197 197 355 355 355 F F F 21 420-439 196 196 196 352 352 352 F F F 22 440-459 199 199 199 361 361 361 F F F 23 460-479 197 197 197 355 355 355 F F F 24 480-499 189 189 189 341 341 341 F F F 25 500-519 195 195 195 348 348 348 F F F 26 520-539 192 192 192 345 345 345 F F F 27 540-559 197 197 197 355 355 355 F F F 28 560-579 199 199 199 361 361 361 F F F 29 580-599 199 199 199 361 361 361 F F F Max gen: 199, Max flush: 361 7:28 Test duration Test completed real 7:27.994778869 user 26.064677303 sys 14.650891068 trap 0.020006940 tflt 0.075413770 dflt 0.084819575 kflt 0.016442525 lock 11:54:51.628893996 slp 36:16.388437759 lat 2.404297182 stop 0.019704108