Turn off color for downstairs dump Removing existing region "/var/tmp/test_repair" Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/test_live_repair/test_repair_dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.24128337 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.015018552 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.014379094 Created 3 regions /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1203 1204 1205 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.34763201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.365752443Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.366961203Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.367057553Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.367070435Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.367088184Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208} {"msg":"Crucible 5825488f-e2ae-483d-9635-d48ec2cc4635 has session id: 99468654-26f4-4eda-9e14-6b3c559a6509","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.36768248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Upstairs opts: Upstairs UUID: 5825488f-e2ae-483d-9635-d48ec2cc4635, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.367754035Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Crucible stats registered with UUID: 5825488f-e2ae-483d-9635-d48ec2cc4635","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.367771184Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:01.367856721Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.368524441Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.368575859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.368595247Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.368777112Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.368895078Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208} {"msg":"5825488f-e2ae-483d-9635-d48ec2cc4635 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.368942508Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.368956699Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"client":"0","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.368988769Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"client":"1","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.369013274Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"client":"2","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.369027355Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.369042226Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.369056497Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"ds_connection connected from Ok(127.0.0.1:50418)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.369302432Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"ds_connection connected from Ok(127.0.0.1:64045)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.369329965Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"ds_connection connected from Ok(127.0.0.1:62005)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.369346325Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.371158246Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"client":"2","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.371195742Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"client":"0","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.37121539Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"client":"1","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.371975382Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372025721Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372184231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372200421Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372217011Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372231701Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372246552Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372262912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372277652Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372292343Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372308703Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372322344Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372336475Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372349367Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372367626Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"":"downstairs","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372382267Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372395019Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"client":"0","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.37240905Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"client":"1","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372422781Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"client":"2","":"downstairs","session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372437332Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372450544Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"5825488f-e2ae-483d-9635-d48ec2cc4635 is now active with session: 99468654-26f4-4eda-9e14-6b3c559a6509","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372465394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372478106Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208,"session_id":"99468654-26f4-4eda-9e14-6b3c559a6509"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372491308Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208} {"msg":"Activated sub_volume 5825488f-e2ae-483d-9635-d48ec2cc4635","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:01.372502631Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Fill test Read and Verify all blocks (0..600 range:false) Wrote out file "/tmp/test_live_repair/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Read 88 Done Done Done false 1005 Acked Read 256 Done Done Done false 1006 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1006 1005 1004 1003 1002 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1007 1006 1005 1004 1003 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:05.439363661Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1208} Begin loop 1 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.466069388Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.47776278Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478092433Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.47812853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478140892Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478158641Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210} {"msg":"Crucible 13ed8a4f-3a48-4c96-a1b0-ff0f645a4f2c has session id: d74b03c5-81df-4434-9836-1b0f4a773240","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478513428Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Upstairs opts: Upstairs UUID: 13ed8a4f-3a48-4c96-a1b0-ff0f645a4f2c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478547767Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Crucible stats registered with UUID: 13ed8a4f-3a48-4c96-a1b0-ff0f645a4f2c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478568384Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:05.478581945Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478738457Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"io task","client":"0","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478772285Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"io task","client":"2","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478787916Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"io task","client":"1","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478877659Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"task":"crutest"} {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478893129Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210} {"msg":"13ed8a4f-3a48-4c96-a1b0-ff0f645a4f2c active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478912287Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478925049Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"client":"0","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478939011Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"client":"1","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478952892Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"client":"2","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478966343Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"io task","client":"2","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.478980794Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"io task","client":"0","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.479008467Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"io task","client":"1","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"ds_connection connected from Ok(127.0.0.1:34555)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.479024377Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"io task","client":"1","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"ds_connection connected from Ok(127.0.0.1:47086)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.479039448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"io task","client":"2","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"ds_connection connected from Ok(127.0.0.1:35403)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.479054598Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"io task","client":"0","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.47984675Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"client":"1","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.479907862Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"client":"0","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.47992627Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"client":"2","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480209912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480242781Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480304083Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480321022Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480337922Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480352552Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480367943Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480384442Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480399553Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480414314Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480430433Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480444135Z"Wait for a query_work_queue command to finish before sending IO ,"hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480464122Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480477384Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480490756Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"":"downstairs","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480507215Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480520027Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"client":"0","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480534178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"client":"1","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480548179Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"client":"2","":"downstairs","session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480565359Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.4805787Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"13ed8a4f-3a48-4c96-a1b0-ff0f645a4f2c is now active with session: d74b03c5-81df-4434-9836-1b0f4a773240","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480593571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480606143Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210,"session_id":"d74b03c5-81df-4434-9836-1b0f4a773240"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480619175Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210} {"msg":"Activated sub_volume 13ed8a4f-3a48-4c96-a1b0-ff0f645a4f2c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:05.480630658Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 324 len 32768 data: 2 2 2 2 2 2 2 2 02/30 Read block 191 len 40960 03/30 Write block 472 len 4096 data: 2 04/30 Write block 515 len 12288 data: 2 2 2 05/30 Write block 425 len 8192 data: 2 2 06/30 Read block 479 len 16384 07/30 Write block 131 len 4096 data: 2 08/30 Write block 310 len 36864 data: 2 2 2 2 2 2 2 2 2 09/30 Flush 10/30 Flush 11/30 Write block 27 len 4096 data: 2 12/30 Read block 59 len 28672 13/30 Read block 310 len 36864 14/30 Read block 151 len 32768 15/30 Flush 16/30 Write block 515 len 8192 data: 3 3 17/30 Read block 416 len 32768 18/30 Flush 19/30 Flush 20/30 Write block 480 len 8192 data: 2 2 21/30 Flush 22/30 Write block 508 len 12288 data: 2 2 2 23/30 Read block 300 len 12288 24/30 Read block 30 len 32768 25/30 Write block 394 len 36864 data: 2 2 2 2 2 2 2 2 2 26/30 Write block 282 len 4096 data: 2 27/30 Write block 83 len 32768 data: 2 2 2 2 2 2 2 2 28/30 Read block 477 len 8192 29/30 Read block 89 len 8192 30/30 Read block 290 len 8192 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Write 1 Done Done Sent false 1003 Acked Write 3 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Read 4 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 3 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Read 2 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 28 28 Done 30 30 2 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:05.491773055Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1210} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 2 2 1 4 4 1 F F F <--- 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 1 1 1 1 1 1 T T F <--- 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 1 2 2 1 F F F <--- 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 1 1 1 1 1 1 F F F 14 280-299 1 1 1 1 1 1 T T F <--- 15 300-319 2 2 1 2 2 1 F F F <--- 16 320-339 2 2 1 2 2 1 F F T <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 1 1 1 1 1 1 F F F 19 380-399 1 1 1 1 1 1 T T F <--- 20 400-419 1 1 1 1 1 1 T T F <--- 21 420-439 2 2 1 2 2 1 F F F <--- 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 1 2 2 1 F F F <--- 24 480-499 2 2 1 7 7 1 F F F <--- 25 500-519 2 2 1 5 5 1 T T F <--- 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 2, Max flush: 7 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 2 2 2 4 4 4 F F F 2 040-059 1 1 1 1 1 1 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 3 3 3 8 8 8 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 1 1 1 1 1 1 F F F 14 280-299 3 3 3 8 8 8 F F F 15 300-319 2 2 2 2 2 2 F F F 16 320-339 2 2 2 2 2 2 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 1 1 1 1 1 1 F F F 19 380-399 3 3 3 8 8 8 F F F 20 400-419 3 3 3 8 8 8 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 2 2 2 2 2 2 F F F 24 480-499 2 2 2 7 7 7 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 3, Max flush: 8 Begin loop 2 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.963456995Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:09.974055254Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.974113808Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.984482941Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.984778356Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.984818081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.984830023Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.984847922Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218} {"msg":"Crucible 04ed8a6b-d281-404d-b9f8-b2c71f0f902f has session id: d1131a11-e421-47ef-9177-c33c7584688b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985190577Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Upstairs opts: Upstairs UUID: 04ed8a6b-d281-404d-b9f8-b2c71f0f902f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985225645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Crucible stats registered with UUID: 04ed8a6b-d281-404d-b9f8-b2c71f0f902f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985241455Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:09.985254996Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985413377Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"io task","client":"2","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985447595Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"io task","client":"0","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985463385Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"io task","client":"1","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985564192Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"task":"crutest"} {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985579462Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218} {"msg":"04ed8a6b-d281-404d-b9f8-b2c71f0f902f active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985600799Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985613521Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"client":"0","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985627502Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"client":"1","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985642433Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"client":"2","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985691872Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"io task","client":"2","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985716476Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"io task","client":"0","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985743349Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"io task","client":"1","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"ds_connection connected from Ok(127.0.0.1:36771)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.98575882Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"io task","client":"0","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"ds_connection connected from Ok(127.0.0.1:58499)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.98577386Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"io task","client":"2","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"ds_connection connected from Ok(127.0.0.1:46701)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.985788811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"io task","client":"1","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.986520391Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"client":"0","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.986549163Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"client":"2","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.986566462Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"client":"1","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.986924277Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.986962293Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"[0]R flush_numbers[0..12]: [1, 4, 1, 1, 8, 1, 2, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.98701577Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"[0]R generation[0..12]: [1, 2, 1, 1, 3, 1, 2, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987032999Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987049529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"[1]R flush_numbers[0..12]: [1, 4, 1, 1, 8, 1, 2, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987064659Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"[1]R generation[0..12]: [1, 2, 1, 1, 3, 1, 2, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.98707964Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.9870957Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"[2]R flush_numbers[0..12]: [1, 4, 1, 1, 8, 1, 2, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987113528Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"[2]R generation[0..12]: [1, 2, 1, 1, 3, 1, 2, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987128569Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:01:09.987144679Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987164007Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987177808Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Next flush: 9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.98719094Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987204271Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"":"downstairs","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987218382Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987230935Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"client":"0","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987245076Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"client":"1","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:01:09.987258737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"client":"2","":"downstairs","session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987275107Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987288568Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"04ed8a6b-d281-404d-b9f8-b2c71f0f902f is now active with session: d1131a11-e421-47ef-9177-c33c7584688b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987303749Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987316451Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218,"session_id":"d1131a11-e421-47ef-9177-c33c7584688b"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987330982Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218} {"msg":"Activated sub_volume 04ed8a6b-d281-404d-b9f8-b2c71f0f902f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:09.987342584Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 3 len 4096 data: 2 03/30 Read block 500 len 28672 04/30 Write block 366 len 16384 data: 2 2 2 2 05/30 Read block 61 len 40960 06/30 Write block 444 len 28672 data: 2 2 2 2 2 2 2 07/30 Read block 591 len 12288 08/30 Write block 596 len 8192 data: 2 2 09/30 Write block 68 len 32768 data: 2 2 2 2 2 2 2 2 10/30 Write block 215 len 40960 data: 2 2 2 2 2 2 2 2 2 2 11/30 Write block 583 len 36864 data: 2 2 2 2 2 2 2 2 2 12/30 Read block 384 len 36864 13/30 Write block 413 len 28672 data: 2 2 2 2 2 2 2 14/30 Read block 424 len 40960 15/30 Read block 258 len 12288 16/30 Write block 102 len 20480 data: 2 2 2 2 2 17/30 Write block 322 len 36864 data: 2 2 3 3 3 3 3 3 3 18/30 Read block 357 len 36864 19/30 Read block 27 len 36864 20/30 Read block 266 len 20480 21/30 Read block 185 len 36864 22/30 Write block 55 len 24576 data: 2 2 2 2 2 2 23/30 Write block 395 len 12288 data: 3 3 3 24/30 Read block 257 len 20480 25/30 Write block 366 len 16384 data: 3 3 3 3 26/30 Read block 121 len 24576 27/30 Read block 124 len 16384 28/30 Write block 265 len 24576 data: 2 2 2 2 2 2 29/30 Read block 30 len 16384 30/30 Read block 446 len 20480 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Sent Done Done false 1001 Acked Write 1 Sent Done Done false 1002 Acked Read 7 Sent Done Done false 1003 Acked Write 4 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Write 7 Sent Done Done false 1006 Acked Read 3 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Read 5 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 0 31 Done 0 29 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:09.999864425Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1218} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F T T <--- 1 020-039 2 2 2 4 4 4 F F F 2 040-059 1 1 1 1 1 1 F T T <--- 3 060-079 1 1 1 1 1 1 F T T <--- 4 080-099 3 3 3 8 8 8 F F F 5 100-119 1 1 1 1 1 1 F T T <--- 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 1 1 1 1 1 F T T <--- 11 220-239 1 1 1 1 1 1 F T T <--- 12 240-259 1 1 1 1 1 1 F F F 13 260-279 1 1 1 1 1 1 F T T <--- 14 280-299 3 3 3 8 8 8 F F F 15 300-319 2 2 2 2 2 2 F F F 16 320-339 2 2 2 2 2 2 F T T <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 1 1 1 1 1 1 F T T <--- 19 380-399 3 3 3 8 8 8 F T T <--- 20 400-419 3 3 3 8 8 8 F T T <--- 21 420-439 2 2 2 2 2 2 F F F 22 440-459 1 1 1 1 1 1 F T T <--- 23 460-479 2 2 2 2 2 2 F F F 24 480-499 2 2 2 7 7 7 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 1 1 1 1 1 1 F T T <--- Max gen: 3, Max flush: 8 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 2 2 2 4 4 4 F F F 2 040-059 4 4 4 9 9 9 F F F 3 060-079 4 4 4 9 9 9 F F F 4 080-099 3 3 3 8 8 8 F F F 5 100-119 4 4 4 9 9 9 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 4 4 4 9 9 9 F F F 11 220-239 4 4 4 9 9 9 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 4 4 4 9 9 9 F F F 14 280-299 3 3 3 8 8 8 F F F 15 300-319 2 2 2 2 2 2 F F F 16 320-339 4 4 4 9 9 9 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 4 4 4 9 9 9 F F F 19 380-399 4 4 4 9 9 9 F F F 20 400-419 4 4 4 9 9 9 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 4 4 4 9 9 9 F F F 23 460-479 2 2 2 2 2 2 F F F 24 480-499 2 2 2 7 7 7 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 4, Max flush: 9 Begin loop 3 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.326343526Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.338146609Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.338456114Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.338492601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.338505024Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.338522812Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226} {"msg":"Crucible 3486061b-6684-4b52-8ab8-ab6c219cbf96 has session id: 9f5ffce8-1fc8-4d62-8d8e-b32f1486084f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.33887667Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Upstairs opts: Upstairs UUID: 3486061b-6684-4b52-8ab8-ab6c219cbf96, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.338910899Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Crucible stats registered with UUID: 3486061b-6684-4b52-8ab8-ab6c219cbf96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.338927198Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:14.33894055Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339102898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"io task","client":"0","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339127632Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"io task","client":"1","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339142293Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"io task","client":"2","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339260569Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"task":"crutest"} {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339276239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226} {"msg":"3486061b-6684-4b52-8ab8-ab6c219cbf96 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339293728Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.33930662Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"client":"0","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339320861Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"client":"1","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339335022Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"client":"2","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339349013Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"io task","client":"2","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339363054Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"io task","client":"0","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339377116Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"io task","client":"1","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"ds_connection connected from Ok(127.0.0.1:34846)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339411364Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"io task","client":"0","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"ds_connection connected from Ok(127.0.0.1:35079)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339427874Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"io task","client":"2","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"ds_connection connected from Ok(127.0.0.1:65188)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.339505735Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"io task","client":"1","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340155676Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"client":"0","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340204755Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"client":"1","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340393047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"client":"2","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.34053252Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340570566Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"[0]R flush_numbers[0..12]: [9, 4, 9, 9, 8, 9, 2, 1, 1, 1, 9, 9]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340726448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"[0]R generation[0..12]: [4, 2, 4, 4, 3, 4, 2, 1, 1, 1, 4, 4]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340742817Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340759637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"[1]R flush_numbers[0..12]: [9, 4, 9, 9, 8, 9, 2, 1, 1, 1, 9, 9]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340774557Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"[1]R generation[0..12]: [4, 2, 4, 4, 3, 4, 2, 1, 1, 1, 4, 4]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340789508Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340805848Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"[2]R flush_numbers[0..12]: [9, 4, 9, 9, 8, 9, 2, 1, 1, 1, 9, 9]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340820758Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"[2]R generation[0..12]: [4, 2, 4, 4, 3, 4, 2, 1, 1, 1, 4, 4]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340835289Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340851349Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Max found gen is 5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340864991Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Generation requested: 6 >= found:5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340884568Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Next flush: 10","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340911491Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"":"downstairs","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340927341Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340940123Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"client":"0","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340960381Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"client":"1","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:01:14.340974272Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"client":"2","":"downstairs","session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.340990042Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.341003753Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"3486061b-6684-4b52-8ab8-ab6c219cbf96 is now active with session: 9f5ffce8-1fc8-4d62-8d8e-b32f1486084f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.341018504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.341031436Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226,"session_id":"9f5ffce8-1fc8-4d62-8d8e-b32f1486084f"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.341044838Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226} {"msg":"Activated sub_volume 3486061b-6684-4b52-8ab8-ab6c219cbf96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:14.34105627Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 199 len 20480 data: 2 2 2 2 2 02/30 Write block 282 len 20480 data: 3 2 2 2 2 03/30 Read block 238 len 36864 04/30 Read block 440 len 20480 05/30 Read block 233 len 40960 06/30 Flush 07/30 Write block 456 len 28672 data: 2 2 2 2 2 2 2 08/30 Write block 233 len 24576 data: 2 2 2 2 2 2 09/30 Write block 182 len 40960 data: 2 2 2 2 2 2 2 2 2 2 10/30 Write block 346 len 32768 data: 2 2 2 2 2 2 2 2 11/30 Read block 376 len 16384 12/30 Read block 129 len 16384 13/30 Write block 336 len 4096 data: 2 14/30 Write block 194 len 40960 data: 2 2 2 2 2 3 3 3 3 3 15/30 Read block 340 len 28672 16/30 Write block 22 len 4096 data: 2 17/30 Write block 117 len 16384 data: 2 2 2 2 18/30 Read block 183 len 36864 19/30 Write block 243 len 36864 data: 2 2 2 2 2 2 2 2 2 20/30 Read block 159 len 4096 21/30 Write block 556 len 20480 data: 2 2 2 2 2 22/30 Write block 85 len 36864 data: 3 3 3 3 3 3 2 2 2 23/30 Write block 242 len 28672 data: 2 3 3 3 3 3 3 24/30 Read block 63 len 28672 25/30 Read block 322 len 4096 26/30 Write block 525 len 40960 data: 2 2 2 2 2 2 2 2 2 2 27/30 Write block 554 len 40960 data: 2 2 3 3 3 3 3 2 2 2 28/30 Read block 141 len 32768 29/30 Read block 25 len 16384 30/30 Write block 315 len 4096 data: 3 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Sent false 1001 Acked Write 5 Done Done Sent false 1002 Acked Read 9 Done Done Sent false 1003 Acked Read 5 Done Done Sent false 1004 Acked Read 10 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 7 Done Done Sent false 1007 Acked Write 6 Done Done Sent false 1008 Acked Write 10 Done Done Sent false 1009 Acked Write 8 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Read 4 Done Done Sent false 1012 Acked Write 1 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 1 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1005)) Some(JobId(1005)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:14.353280658Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1226} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 2 2 2 4 4 4 T T F <--- 2 040-059 4 4 4 9 9 9 F F F 3 060-079 4 4 4 9 9 9 F F F 4 080-099 3 3 3 8 8 8 T T F <--- 5 100-119 4 4 4 9 9 9 T T F <--- 6 120-139 2 2 2 2 2 2 T T F <--- 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 6 6 1 10 10 1 T T F <--- 10 200-219 6 6 4 10 10 9 T T F <--- 11 220-239 4 4 4 9 9 9 T T F <--- 12 240-259 1 1 1 1 1 1 T T F <--- 13 260-279 4 4 4 9 9 9 F F F 14 280-299 6 6 3 10 10 8 F F F <--- 15 300-319 2 2 2 2 2 2 T T F <--- 16 320-339 4 4 4 9 9 9 T T F <--- 17 340-359 1 1 1 1 1 1 T T F <--- 18 360-379 4 4 4 9 9 9 F F F 19 380-399 4 4 4 9 9 9 F F F 20 400-419 4 4 4 9 9 9 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 4 4 4 9 9 9 T T F <--- 23 460-479 2 2 2 2 2 2 T T F <--- 24 480-499 2 2 2 7 7 7 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 1 1 1 1 1 1 T T F <--- 27 540-559 1 1 1 1 1 1 T T F <--- 28 560-579 1 1 1 1 1 1 T T F <--- 29 580-599 4 4 4 9 9 9 F F F Max gen: 6, Max flush: 10 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 7 7 7 11 11 11 F F F 2 040-059 4 4 4 9 9 9 F F F 3 060-079 4 4 4 9 9 9 F F F 4 080-099 7 7 7 11 11 11 F F F 5 100-119 7 7 7 11 11 11 F F F 6 120-139 7 7 7 11 11 11 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 11 11 11 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 7 7 7 11 11 11 F F F 12 240-259 7 7 7 11 11 11 F F F 13 260-279 4 4 4 9 9 9 F F F 14 280-299 6 6 6 10 10 10 F F F 15 300-319 7 7 7 11 11 11 F F F 16 320-339 7 7 7 11 11 11 F F F 17 340-359 7 7 7 11 11 11 F F F 18 360-379 4 4 4 9 9 9 F F F 19 380-399 4 4 4 9 9 9 F F F 20 400-419 4 4 4 9 9 9 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 7 7 7 11 11 11 F F F 23 460-479 7 7 7 11 11 11 F F F 24 480-499 2 2 2 7 7 7 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 7 7 7 11 11 11 F F F 27 540-559 7 7 7 11 11 11 F F F 28 560-579 7 7 7 11 11 11 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 7, Max flush: 11 Begin loop 4 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.679397945Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.691003622Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.691324031Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.691363426Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.691376088Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.691393907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234} {"msg":"Crucible 5c8f7467-2362-4bea-acaf-b167f4c80518 has session id: cd871118-59c0-446c-9b39-916470f90619","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.691737611Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Upstairs opts: Upstairs UUID: 5c8f7467-2362-4bea-acaf-b167f4c80518, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.691772369Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Crucible stats registered with UUID: 5c8f7467-2362-4bea-acaf-b167f4c80518","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.691788239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:18.69180186Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.691951276Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"io task","client":"1","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.691986314Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"io task","client":"0","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692002015Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"io task","client":"2","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692109007Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"task":"crutest"} {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692125947Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234} {"msg":"5c8f7467-2362-4bea-acaf-b167f4c80518 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692141317Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692153789Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"client":"0","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.69216797Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"client":"1","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692181981Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"client":"2","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692195513Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"io task","client":"2","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692209974Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"io task","client":"0","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692226174Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"io task","client":"1","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"ds_connection connected from Ok(127.0.0.1:49144)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692301506Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"io task","client":"0","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"ds_connection connected from Ok(127.0.0.1:58517)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.692326091Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"io task","client":"2","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"ds_connection connected from Ok(127.0.0.1:49278)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.69242247Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"io task","client":"1","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693048047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"client":"2","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693086443Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"client":"1","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693103642Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"client":"0","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693440521Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693492408Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"[0]R flush_numbers[0..12]: [9, 11, 9, 9, 11, 11, 11, 1, 1, 11, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693538749Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"[0]R generation[0..12]: [4, 7, 4, 4, 7, 7, 7, 1, 1, 7, 7, 7]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693554829Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693571348Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"[1]R flush_numbers[0..12]: [9, 11, 9, 9, 11, 11, 11, 1, 1, 11, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693586729Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"[1]R generation[0..12]: [4, 7, 4, 4, 7, 7, 7, 1, 1, 7, 7, 7]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693601639Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693618099Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"[2]R flush_numbers[0..12]: [9, 11, 9, 9, 11, 11, 11, 1, 1, 11, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693636178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"[2]R generation[0..12]: [4, 7, 4, 4, 7, 7, 7, 1, 1, 7, 7, 7]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693651488Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693667928Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Max found gen is 8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693681469Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.69369556Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Next flush: 12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693709002Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO time":"2026-03-14T05:01:18.693722373Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"":"downstairs","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693742151Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693755063Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"client":"0","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693769164Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"client":"1","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693782875Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"client":"2","":"downstairs","session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693796967Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693810288Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"5c8f7467-2362-4bea-acaf-b167f4c80518 is now active with session: cd871118-59c0-446c-9b39-916470f90619","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693825019Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.69383908Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234,"session_id":"cd871118-59c0-446c-9b39-916470f90619"} {"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693853861Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234} {"msg":"Activated sub_volume 5c8f7467-2362-4bea-acaf-b167f4c80518","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:18.693865264Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 132 len 16384 data: 2 2 2 2 02/30 Read block 86 len 40960 03/30 Write block 360 len 12288 data: 2 2 2 04/30 Read block 146 len 40960 05/30 Read block 504 len 12288 06/30 Read block 184 len 20480 07/30 Write block 307 len 28672 data: 2 2 2 3 3 3 3 08/30 Write block 308 len 32768 data: 3 3 4 4 4 4 3 4 09/30 Write block 455 len 12288 data: 2 3 3 10/30 Read block 325 len 16384 11/30 Read block 360 len 8192 12/30 Write block 95 len 12288 data: 2 2 2 13/30 Write block 84 len 12288 data: 3 4 4 14/30 Write block 532 len 20480 data: 3 3 3 2 2 15/30 Read block 68 len 28672 16/30 Write block 312 len 20480 data: 5 5 4 5 3 17/30 Read block 292 len 8192 18/30 Read block 205 len 40960 19/30 Read block 217 len 28672 20/30 Write block 561 len 24576 data: 3 3 3 2 2 2 21/30 Read block 147 len 20480 22/30 Write block 248 len 12288 data: 4 3 3 23/30 Read block 4 len 20480 24/30 Write block 407 len 20480 data: 2 2 2 2 2 25/30 Flush 26/30 Write block 560 len 4096 data: 4 27/30 Write block 459 len 8192 data: 3 3 28/30 Read block 133 len 16384 29/30 Write block 478 len 28672 data: 2 2 3 3 2 2 2 30/30 Read block 115 len 32768 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Sent Done false 1001 Acked Read 10 Done Sent Done false 1002 Acked Write 3 Done Sent Done false 1003 Acked Read 10 Done Sent Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Write 8 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Read 2 Done Sent Done false 1011 Acked Write 3 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Write 6 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Write 5 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Write 7 Done Sent Done false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 1 31 Done 30 0 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:18.705365397Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1234} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 7 7 7 11 11 11 F F F 2 040-059 4 4 4 9 9 9 F F F 3 060-079 4 4 4 9 9 9 F F F 4 080-099 8 7 8 12 11 12 F F F <--- 5 100-119 7 7 7 11 11 11 F F F 6 120-139 8 7 8 12 11 12 F F F <--- 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 11 11 11 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 7 7 7 11 11 11 F F F 12 240-259 8 7 8 12 11 12 F F F <--- 13 260-279 4 4 4 9 9 9 F F F 14 280-299 6 6 6 10 10 10 F F F 15 300-319 8 7 8 12 11 12 F F F <--- 16 320-339 7 7 7 11 11 11 F F F 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 4 8 12 9 12 F F F <--- 19 380-399 4 4 4 9 9 9 F F F 20 400-419 8 4 8 12 9 12 F F F <--- 21 420-439 2 2 2 2 2 2 F F F 22 440-459 8 7 8 12 11 12 T F T <--- 23 460-479 7 7 7 11 11 11 T F T <--- 24 480-499 2 2 2 7 7 7 T F T <--- 25 500-519 3 3 3 8 8 8 F F F 26 520-539 8 7 8 12 11 12 F F F <--- 27 540-559 7 7 7 11 11 11 F F F 28 560-579 8 7 8 12 11 12 T F T <--- 29 580-599 4 4 4 9 9 9 F F F Max gen: 8, Max flush: 12 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 7 7 7 11 11 11 F F F 2 040-059 4 4 4 9 9 9 F F F 3 060-079 4 4 4 9 9 9 F F F 4 080-099 8 8 8 12 12 12 F F F 5 100-119 7 7 7 11 11 11 F F F 6 120-139 8 8 8 12 12 12 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 11 11 11 F F F 10 200-219 7 7 7 11 11 11 F F F 11 220-239 7 7 7 11 11 11 F F F 12 240-259 8 8 8 12 12 12 F F F 13 260-279 4 4 4 9 9 9 F F F 14 280-299 6 6 6 10 10 10 F F F 15 300-319 8 8 8 12 12 12 F F F 16 320-339 7 7 7 11 11 11 F F F 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 4 4 4 9 9 9 F F F 20 400-419 8 8 8 12 12 12 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 9 9 9 13 13 13 F F F 23 460-479 9 9 9 13 13 13 F F F 24 480-499 9 9 9 13 13 13 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 8 8 8 12 12 12 F F F 27 540-559 7 7 7 11 11 11 F F F 28 560-579 9 9 9 13 13 13 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 9, Max flush: 13 Begin loop 5 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.016238642Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:23.026846776Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.026895006Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.037293041Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.037596101Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.037635246Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.037647109Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.037664888Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242} {"msg":"Crucible 4f5599cd-2804-424a-9b29-13d1c6104765 has session id: 30c2ce0b-e752-4050-a896-34daa7565468","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038015857Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Upstairs opts: Upstairs UUID: 4f5599cd-2804-424a-9b29-13d1c6104765, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038050196Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Crucible stats registered with UUID: 4f5599cd-2804-424a-9b29-13d1c6104765","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038066285Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:23.038079717Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038247242Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"io task","client":"0","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038284478Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"io task","client":"2","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038300448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"io task","client":"1","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038398577Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"task":"crutest"} {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038412738Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242} {"msg":"4f5599cd-2804-424a-9b29-13d1c6104765 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038430087Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038442909Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"client":"0","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.03845673Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"client":"1","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038470931Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"client":"2","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038484992Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"io task","client":"2","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038499443Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"io task","client":"0","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038513354Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"io task","client":"1","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"ds_connection connected from Ok(127.0.0.1:33889)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038528175Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"io task","client":"2","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"ds_connection connected from Ok(127.0.0.1:38585)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038543406Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"io task","client":"1","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"ds_connection connected from Ok(127.0.0.1:55501)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.038558506Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"io task","client":"0","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039313452Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"client":"1","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039349689Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"client":"0","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039369287Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"client":"2","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039709423Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039725603Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"[0]R flush_numbers[0..12]: [9, 11, 9, 9, 12, 11, 12, 1, 1, 11, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039759701Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"[0]R generation[0..12]: [4, 7, 4, 4, 8, 7, 8, 1, 1, 7, 7, 7]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039775631Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039792331Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"[1]R flush_numbers[0..12]: [9, 11, 9, 9, 12, 11, 12, 1, 1, 11, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039807731Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"[1]R generation[0..12]: [4, 7, 4, 4, 8, 7, 8, 1, 1, 7, 7, 7]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039822582Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039838732Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"[2]R flush_numbers[0..12]: [9, 11, 9, 9, 12, 11, 12, 1, 1, 11, 11, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039853832Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"[2]R generation[0..12]: [4, 7, 4, 4, 8, 7, 8, 1, 1, 7, 7, 7]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039870102Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039887041Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039900833Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.03992079Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Next flush: 14","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.03993673Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039950281Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"":"downstairs","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039964622Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039977334Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"client":"0","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.039991765Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"client":"1","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.040005946Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"client":"2","":"downstairs","session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.040021577Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.040035398Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"4f5599cd-2804-424a-9b29-13d1c6104765 is now active with session: 30c2ce0b-e752-4050-a896-34daa7565468","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.040050299Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.040062821Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242,"session_id":"30c2ce0b-e752-4050-a896-34daa7565468"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.040075943Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242} {"msg":"Activated sub_volume 4f5599cd-2804-424a-9b29-13d1c6104765","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:23.040087195Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 215 len 40960 data: 3 3 3 3 3 3 3 3 3 3 02/30 Write block 108 len 4096 data: 2 03/30 Read block 482 len 24576 04/30 Write block 262 len 12288 data: 2 2 2 05/30 Write block 423 len 12288 data: 2 2 3 06/30 Read block 215 len 36864 07/30 Read block 122 len 24576 08/30 Read block 474 len 12288 09/30 Write block 74 len 24576 data: 3 3 2 2 2 2 10/30 Write block 330 len 8192 data: 4 3 11/30 Write block 109 len 20480 data: 2 2 2 2 2 12/30 Write block 273 len 32768 data: 2 2 2 2 2 2 2 2 13/30 Write block 101 len 40960 data: 2 3 3 3 3 3 2 3 3 3 14/30 Flush 15/30 Write block 60 len 28672 data: 3 2 2 2 2 2 2 16/30 Write block 530 len 32768 data: 3 3 4 4 4 3 3 2 17/30 Read block 400 len 40960 18/30 Read block 165 len 20480 19/30 Read block 472 len 36864 20/30 Write block 455 len 20480 data: 3 4 4 3 4 21/30 Read block 201 len 8192 22/30 Write block 301 len 32768 data: 2 2 2 2 2 2 3 4 23/30 Read block 76 len 32768 24/30 Read block 313 len 4096 25/30 Read block 456 len 16384 26/30 Read block 360 len 20480 27/30 Write block 431 len 40960 data: 2 2 2 2 2 2 2 2 2 2 28/30 Write block 108 len 28672 data: 4 4 4 3 3 3 2 29/30 Write block 122 len 28672 data: 2 2 2 2 2 2 2 30/30 Write block 227 len 8192 data: 2 2 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Write 3 Done Done Done false 1005 Acked Read 9 Done Done Done false 1006 Acked Read 6 Done Done Done false 1007 Acked Read 3 Done Done Done false 1008 Acked Write 6 Done Done Done false 1009 Acked Write 2 Done Done Done false 1010 Acked Write 5 Done Done Done false 1011 Acked Write 8 Done Done Done false 1012 Acked Write 10 Done Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Read 10 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 17 2 2 21 Done 13 28 28 69 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1013)) Some(JobId(1013)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:23.051754304Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1242} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 7 7 7 11 11 11 F F F 2 040-059 4 4 4 9 9 9 F F F 3 060-079 4 10 10 9 14 14 T T T <--- 4 080-099 8 8 8 12 12 12 F F F 5 100-119 7 10 10 11 14 14 T T T <--- 6 120-139 8 8 8 12 12 12 F T T <--- 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 11 11 11 F F F 10 200-219 7 10 10 11 14 14 T F F <--- 11 220-239 7 10 10 11 14 14 T T T <--- 12 240-259 8 8 8 12 12 12 F F F 13 260-279 4 10 10 9 14 14 T F F <--- 14 280-299 6 10 10 10 14 14 T F F <--- 15 300-319 8 8 8 12 12 12 F T T <--- 16 320-339 7 10 10 11 14 14 T F F <--- 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 4 4 4 9 9 9 F F F 20 400-419 8 8 8 12 12 12 F F F 21 420-439 2 10 10 2 14 14 T T T <--- 22 440-459 9 9 9 13 13 13 F T T <--- 23 460-479 9 9 9 13 13 13 F F F 24 480-499 9 9 9 13 13 13 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 8 8 8 12 12 12 F T T <--- 27 540-559 7 7 7 11 11 11 F F F 28 560-579 9 9 9 13 13 13 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 10, Max flush: 14 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 7 7 7 11 11 11 F F F 2 040-059 4 4 4 9 9 9 F F F 3 060-079 11 11 11 15 15 15 F F F 4 080-099 8 8 8 12 12 12 F F F 5 100-119 11 11 11 15 15 15 F F F 6 120-139 11 11 11 15 15 15 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 11 11 11 F F F 10 200-219 10 10 10 14 14 14 F F F 11 220-239 11 11 11 15 15 15 F F F 12 240-259 8 8 8 12 12 12 F F F 13 260-279 10 10 10 14 14 14 F F F 14 280-299 10 10 10 14 14 14 F F F 15 300-319 11 11 11 15 15 15 F F F 16 320-339 10 10 10 14 14 14 F F F 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 4 4 4 9 9 9 F F F 20 400-419 8 8 8 12 12 12 F F F 21 420-439 11 11 11 15 15 15 F F F 22 440-459 11 11 11 15 15 15 F F F 23 460-479 9 9 9 13 13 13 F F F 24 480-499 9 9 9 13 13 13 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 11 11 11 15 15 15 F F F 27 540-559 7 7 7 11 11 11 F F F 28 560-579 9 9 9 13 13 13 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 11, Max flush: 15 Begin loop 6 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.376697329Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.388507067Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.388825877Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.388864473Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.388880892Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.388899191Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250} {"msg":"Crucible 6fb17fe7-25af-4d28-bf49-b82d40f307eb has session id: 59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389211944Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Upstairs opts: Upstairs UUID: 6fb17fe7-25af-4d28-bf49-b82d40f307eb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389238937Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Crucible stats registered with UUID: 6fb17fe7-25af-4d28-bf49-b82d40f307eb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389253528Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:27.38926683Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389442889Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"io task","client":"2","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389477717Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"io task","client":"0","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389493217Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"io task","client":"1","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.3896004Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"task":"crutest"} {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389614421Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250} {"msg":"6fb17fe7-25af-4d28-bf49-b82d40f307eb active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389631151Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389644193Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"client":"0","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389658434Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"client":"1","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389672445Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"client":"2","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389686746Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"io task","client":"2","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389703016Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"io task","client":"0","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389717227Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"io task","client":"1","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"ds_connection connected from Ok(127.0.0.1:57397)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389790071Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"io task","client":"1","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"ds_connection connected from Ok(127.0.0.1:44741)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389817793Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"io task","client":"0","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"ds_connection connected from Ok(127.0.0.1:51311)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.389877946Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"io task","client":"2","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.390527358Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"client":"1","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.390588989Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"client":"0","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.390607717Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"client":"2","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.390895976Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.390939179Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"[0]R flush_numbers[0..12]: [9, 11, 9, 15, 12, 15, 15, 1, 1, 11, 14, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391010394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"[0]R generation[0..12]: [4, 7, 4, 11, 8, 11, 11, 1, 1, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391027084Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391044143Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"[1]R flush_numbers[0..12]: [9, 11, 9, 15, 12, 15, 15, 1, 1, 11, 14, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391059643Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"[1]R generation[0..12]: [4, 7, 4, 11, 8, 11, 11, 1, 1, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391077142Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391093852Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"[2]R flush_numbers[0..12]: [9, 11, 9, 15, 12, 15, 15, 1, 1, 11, 14, 15]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391109072Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"[2]R generation[0..12]: [4, 7, 4, 11, 8, 11, 11, 1, 1, 7, 10, 11]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391123713Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391139983Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Wait for a query_work_queue command to finish before sending IO Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391159041Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Generation requested: 12 >= found:12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391173472Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Next flush: 16","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391186813Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391200115Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"":"downstairs","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391214296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391226968Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"client":"0","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391241059Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"client":"1","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.39125513Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"client":"2","":"downstairs","session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391272189Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391285431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"6fb17fe7-25af-4d28-bf49-b82d40f307eb is now active with session: 59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391302001Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391314683Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250,"session_id":"59f173b4-0e7b-42b4-b37c-2dd8ea00eaa3"} {"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391328014Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250} {"msg":"Activated sub_volume 6fb17fe7-25af-4d28-bf49-b82d40f307eb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:27.391339587Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 479 len 8192 data: 3 4 02/30 Read block 375 len 16384 03/30 Write block 303 len 4096 data: 3 04/30 Write block 547 len 36864 data: 2 2 2 2 2 2 2 3 3 05/30 Write block 546 len 16384 data: 2 3 3 3 06/30 Write block 464 len 12288 data: 2 2 2 07/30 Read block 580 len 4096 08/30 Flush 09/30 Write block 284 len 8192 data: 3 3 10/30 Read block 531 len 24576 11/30 Read block 205 len 4096 12/30 Write block 487 len 24576 data: 2 2 2 2 2 2 13/30 Read block 221 len 4096 14/30 Write block 589 len 4096 data: 3 15/30 Read block 297 len 28672 16/30 Read block 446 len 28672 17/30 Read block 465 len 4096 18/30 Read block 524 len 8192 19/30 Write block 256 len 24576 data: 2 2 2 2 2 2 20/30 Read block 324 len 36864 21/30 Flush 22/30 Flush 23/30 Write block 575 len 40960 data: 2 2 2 2 2 2 2 2 3 3 24/30 Write block 451 len 20480 data: 2 2 2 2 4 25/30 Read block 300 len 28672 26/30 Read block 496 len 36864 27/30 Write block 467 len 32768 data: 2 2 2 2 2 3 2 2 28/30 Write block 222 len 16384 data: 4 4 4 2 29/30 Write block 90 len 28672 data: 4 3 3 3 2 3 3 30/30 Read block 156 len 12288 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 4 Done Done Sent false 1002 Acked Write 1 Done Done Sent false 1003 Acked Write 9 Done Done Sent false 1004 Acked Write 4 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 2 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Read 9 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Read 7 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Write 7 Done Done Sent false 1029 Acked Read 3 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 29 29 Done 30 30 1 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:27.402563694Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1250} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 7 7 7 11 11 11 F F F 2 040-059 4 4 4 9 9 9 F F F 3 060-079 11 11 11 15 15 15 F F F 4 080-099 8 8 8 12 12 12 T T F <--- 5 100-119 11 11 11 15 15 15 F F F 6 120-139 11 11 11 15 15 15 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 11 11 11 F F F 10 200-219 10 10 10 14 14 14 F F F 11 220-239 11 11 11 15 15 15 T T F <--- 12 240-259 12 12 8 17 17 12 F F F <--- 13 260-279 12 12 10 17 17 14 F F F <--- 14 280-299 12 12 10 17 17 14 F F F <--- 15 300-319 12 12 11 16 16 15 F F F <--- 16 320-339 10 10 10 14 14 14 F F F 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 4 4 4 9 9 9 F F F 20 400-419 8 8 8 12 12 12 F F F 21 420-439 11 11 11 15 15 15 F F F 22 440-459 11 11 11 15 15 15 T T F <--- 23 460-479 12 12 9 16 16 13 T T T <--- 24 480-499 12 12 9 17 17 13 F F T <--- 25 500-519 3 3 3 8 8 8 F F F 26 520-539 11 11 11 15 15 15 F F F 27 540-559 12 12 7 16 16 11 F F F <--- 28 560-579 9 9 9 13 13 13 T T F <--- 29 580-599 12 12 4 17 17 9 T T F <--- Max gen: 12, Max flush: 17 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 7 7 7 11 11 11 F F F 2 040-059 4 4 4 9 9 9 F F F 3 060-079 11 11 11 15 15 15 F F F 4 080-099 13 13 13 18 18 18 F F F 5 100-119 11 11 11 15 15 15 F F F 6 120-139 11 11 11 15 15 15 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 7 7 7 11 11 11 F F F 10 200-219 10 10 10 14 14 14 F F F 11 220-239 13 13 13 18 18 18 F F F 12 240-259 12 12 12 17 17 17 F F F 13 260-279 12 12 12 17 17 17 F F F 14 280-299 12 12 12 17 17 17 F F F 15 300-319 12 12 12 16 16 16 F F F 16 320-339 10 10 10 14 14 14 F F F 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 4 4 4 9 9 9 F F F 20 400-419 8 8 8 12 12 12 F F F 21 420-439 11 11 11 15 15 15 F F F 22 440-459 13 13 13 18 18 18 F F F 23 460-479 13 13 13 18 18 18 F F F 24 480-499 12 12 12 17 17 17 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 11 11 11 15 15 15 F F F 27 540-559 12 12 12 16 16 16 F F F 28 560-579 13 13 13 18 18 18 F F F 29 580-599 13 13 13 18 18 18 F F F Max gen: 13, Max flush: 18 Begin loop 7 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.716692215Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:31.727290645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.727345201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.737516619Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.737810504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.737850269Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.737862601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.73788037Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258} {"msg":"Crucible 01116b29-596d-47a8-bbf8-5c585d807fe1 has session id: 8fcd3476-8147-448e-856b-aec2e173c6e9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.73824632Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Upstairs opts: Upstairs UUID: 01116b29-596d-47a8-bbf8-5c585d807fe1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738281038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Crucible stats registered with UUID: 01116b29-596d-47a8-bbf8-5c585d807fe1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738296809Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:31.73831038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738463774Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"io task","client":"2","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738488528Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"io task","client":"0","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738503509Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"io task","client":"1","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738616438Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"task":"crutest"} {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738634386Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258} {"msg":"01116b29-596d-47a8-bbf8-5c585d807fe1 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738657162Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738669984Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"client":"0","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738683965Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"client":"1","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738697986Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"client":"2","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738711928Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"io task","client":"2","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738726538Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"io task","client":"0","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.73874064Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"io task","client":"1","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:53873)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738765194Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"io task","client":"2","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:41739)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738780185Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"io task","client":"0","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"ds_connection connected from Ok(127.0.0.1:55371)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.738794985Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"io task","client":"1","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.739500592Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"client":"2","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.739532682Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"client":"1","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.739561704Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"client":"0","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.739888119Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.739913093Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"[0]R flush_numbers[0..12]: [9, 11, 9, 15, 18, 15, 15, 1, 1, 11, 14, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.739988376Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"[0]R generation[0..12]: [4, 7, 4, 11, 13, 11, 11, 1, 1, 7, 10, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740018487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740035286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"[1]R flush_numbers[0..12]: [9, 11, 9, 15, 18, 15, 15, 1, 1, 11, 14, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740050636Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"[1]R generation[0..12]: [4, 7, 4, 11, 13, 11, 11, 1, 1, 7, 10, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740065787Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740082077Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"[2]R flush_numbers[0..12]: [9, 11, 9, 15, 18, 15, 15, 1, 1, 11, 14, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740097307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"[2]R generation[0..12]: [4, 7, 4, 11, 13, 11, 11, 1, 1, 7, 10, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740112647Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:01:31.740129037Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Max found gen is 14","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740148275Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740164835Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Next flush: 19","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740178156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740191398Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"":"downstairs","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740205829Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740218561Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"client":"0","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Transition from WaitQuorum to Active","v":0Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.7402362Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"client":"1","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740250901Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"client":"2","":"downstairs","session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740265162Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740278563Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"01116b29-596d-47a8-bbf8-5c585d807fe1 is now active with session: 8fcd3476-8147-448e-856b-aec2e173c6e9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740294183Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740306925Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258,"session_id":"8fcd3476-8147-448e-856b-aec2e173c6e9"} {"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740320757Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258} {"msg":"Activated sub_volume 01116b29-596d-47a8-bbf8-5c585d807fe1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:31.740332429Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 240 len 12288 data: 2 2 3 02/30 Read block 382 len 4096 03/30 Read block 451 len 28672 04/30 Read block 589 len 12288 05/30 Read block 295 len 28672 06/30 Read block 211 len 24576 07/30 Write block 558 len 16384 data: 4 4 5 4 08/30 Flush 09/30 Write block 443 len 36864 data: 2 3 3 3 3 3 3 3 3 10/30 Read block 268 len 20480 11/30 Read block 33 len 40960 12/30 Flush 13/30 Write block 92 len 20480 data: 4 4 3 4 4 14/30 Read block 507 len 32768 15/30 Read block 568 len 8192 16/30 Read block 8 len 16384 17/30 Read block 323 len 36864 18/30 Read block 92 len 8192 19/30 Read block 379 len 40960 20/30 Write block 291 len 8192 data: 2 2 21/30 Write block 167 len 24576 data: 2 2 2 2 2 2 22/30 Write block 386 len 16384 data: 2 2 2 2 23/30 Flush 24/30 Write block 138 len 28672 data: 2 2 2 2 2 2 2 25/30 Write block 112 len 24576 data: 4 4 3 2 2 3 26/30 Read block 590 len 40960 27/30 Read block 405 len 32768 28/30 Write block 424 len 16384 data: 3 4 3 2 29/30 Read block 528 len 28672 30/30 Read block 598 len 8192 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:22 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1008 Acked Write 9 Done Done Done false 1009 Acked Read 5 Done Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Read 2 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Write 2 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 2 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 20 0 0 20 Done 2 22 22 46 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:31.752661791Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1258} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 7 7 7 11 11 11 F F F 2 040-059 4 4 4 9 9 9 F F F 3 060-079 11 11 11 15 15 15 F F F 4 080-099 13 14 14 18 21 21 F F F <--- 5 100-119 11 11 11 15 15 15 F T T <--- 6 120-139 11 11 11 15 15 15 F T T <--- 7 140-159 1 1 1 1 1 1 F T T <--- 8 160-179 1 14 14 1 21 21 F F F <--- 9 180-199 7 7 7 11 11 11 F F F 10 200-219 10 10 10 14 14 14 F F F 11 220-239 13 13 13 18 18 18 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 12 12 12 17 17 17 F F F 14 280-299 12 14 14 17 21 21 F F F <--- 15 300-319 12 12 12 16 16 16 F F F 16 320-339 10 10 10 14 14 14 F F F 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 4 14 14 9 21 21 F F F <--- 20 400-419 8 8 8 12 12 12 F F F 21 420-439 11 11 11 15 15 15 F T T <--- 22 440-459 13 14 14 18 20 20 T F F <--- 23 460-479 13 13 13 18 18 18 F F F 24 480-499 12 12 12 17 17 17 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 11 11 11 15 15 15 F F F 27 540-559 14 14 14 19 19 19 F F F 28 560-579 14 14 14 19 19 19 F F F 29 580-599 13 13 13 18 18 18 F F F Max gen: 14, Max flush: 21 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 F F F 1 020-039 7 7 7 11 11 11 F F F 2 040-059 4 4 4 9 9 9 F F F 3 060-079 11 11 11 15 15 15 F F F 4 080-099 14 14 14 21 21 21 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 15 15 15 22 22 22 F F F 7 140-159 15 15 15 22 22 22 F F F 8 160-179 14 14 14 21 21 21 F F F 9 180-199 7 7 7 11 11 11 F F F 10 200-219 10 10 10 14 14 14 F F F 11 220-239 13 13 13 18 18 18 F F F 12 240-259 14 14 14 19 19 19 F F F 13 260-279 12 12 12 17 17 17 F F F 14 280-299 14 14 14 21 21 21 F F F 15 300-319 12 12 12 16 16 16 F F F 16 320-339 10 10 10 14 14 14 F F F 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 14 14 14 21 21 21 F F F 20 400-419 8 8 8 12 12 12 F F F 21 420-439 15 15 15 22 22 22 F F F 22 440-459 14 14 14 20 20 20 F F F 23 460-479 13 13 13 18 18 18 F F F 24 480-499 12 12 12 17 17 17 F F F 25 500-519 3 3 3 8 8 8 F F F 26 520-539 11 11 11 15 15 15 F F F 27 540-559 14 14 14 19 19 19 F F F 28 560-579 14 14 14 19 19 19 F F F 29 580-599 13 13 13 18 18 18 F F F Max gen: 15, Max flush: 22 Begin loop 8 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.067758494Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.079580844Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.079888741Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.079927077Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.079939509Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.079957548Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266} {"msg":"Crucible c1635af2-df52-401c-884d-d720365de530 has session id: abb63fcd-4103-4a09-a375-2392854ff7bb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080294736Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Upstairs opts: Upstairs UUID: c1635af2-df52-401c-884d-d720365de530, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080329154Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Crucible stats registered with UUID: c1635af2-df52-401c-884d-d720365de530","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080344674Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:36.080358346Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080509541Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"io task","client":"1","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080544409Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"io task","client":"0","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080559989Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"io task","client":"2","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.08065481Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"task":"crutest"} {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.08066976Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266} {"msg":"c1635af2-df52-401c-884d-d720365de530 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080687799Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080700581Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"client":"0","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080714512Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"client":"1","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080728373Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"client":"2","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080742065Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"io task","client":"2","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080756176Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"io task","client":"0","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080772955Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"io task","client":"1","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:61074)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080791714Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"io task","client":"0","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:64432)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080806594Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"io task","client":"2","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:49704)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.080873382Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"io task","client":"1","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.081526582Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"client":"0","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.081554394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"client":"2","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.081581038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"client":"1","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.08187903Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.081925851Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"[0]R flush_numbers[0..12]: [9, 11, 9, 15, 21, 22, 22, 22, 21, 11, 14, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.081962878Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"[0]R generation[0..12]: [4, 7, 4, 11, 14, 15, 15, 15, 14, 7, 10, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.081979257Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.081995637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"[1]R flush_numbers[0..12]: [9, 11, 9, 15, 21, 22, 22, 22, 21, 11, 14, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082010997Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"[1]R generation[0..12]: [4, 7, 4, 11, 14, 15, 15, 15, 14, 7, 10, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082026278Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082045696Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"[2]R flush_numbers[0..12]: [9, 11, 9, 15, 21, 22, 22, 22, 21, 11, 14, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082061196Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"[2]R generation[0..12]: [4, 7, 4, 11, 14, 15, 15, 15, 14, 7, 10, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082076456Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082092266Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082105968Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":Wait for a query_work_queue command to finish before sending IO "abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082125935Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Next flush: 23","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082139646Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082153158Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"":"downstairs","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082167029Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082179951Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"client":"0","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082193862Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"client":"1","":"downstairs","session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082207844Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"client":"2","":"downstairs","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082224923Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082238095Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"c1635af2-df52-401c-884d-d720365de530 is now active with session: abb63fcd-4103-4a09-a375-2392854ff7bb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082252935Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082267136Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266,"session_id":"abb63fcd-4103-4a09-a375-2392854ff7bb"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082280268Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266} {"msg":"Activated sub_volume c1635af2-df52-401c-884d-d720365de530","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:36.082292091Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 184 len 40960 data: 3 3 3 3 3 3 3 3 2 2 02/30 Read block 72 len 12288 03/30 Read block 191 len 36864 04/30 Write block 216 len 4096 data: 4 05/30 Read block 541 len 20480 06/30 Write block 244 len 32768 data: 4 4 4 4 5 4 4 3 07/30 Write block 35 len 32768 data: 2 2 2 2 2 2 2 2 08/30 Write block 16 len 28672 data: 2 2 2 2 2 2 3 09/30 Write block 487 len 20480 data: 3 3 3 3 3 10/30 Read block 536 len 16384 11/30 Write block 288 len 12288 data: 2 2 2 12/30 Read block 187 len 8192 13/30 Write block 508 len 8192 data: 3 3 14/30 Read block 314 len 28672 15/30 Read block 88 len 40960 16/30 Write block 414 len 40960 data: 3 3 3 3 3 3 2 2 2 3 17/30 Write block 130 len 28672 data: 2 3 3 3 3 3 2 18/30 Read block 366 len 4096 19/30 Read block 494 len 40960 20/30 Write block 12 len 16384 data: 2 2 2 2 21/30 Write block 439 len 32768 data: 3 3 2 2 3 4 4 4 22/30 Read block 375 len 28672 23/30 Read block 492 len 20480 24/30 Write block 435 len 32768 data: 3 3 3 3 4 4 3 3 25/30 Write block 312 len 40960 data: 6 6 5 6 4 3 3 2 2 2 26/30 Read block 32 len 32768 27/30 Read block 134 len 16384 28/30 Read block 369 len 40960 29/30 Read block 576 len 32768 30/30 Write block 487 len 12288 data: 4 4 4 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Write 8 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Read 2 Done Sent Done false 1012 Acked Write 2 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Read 10 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 1 28 Done 29 4 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:36.094643509Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1266} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 9 9 9 T F T <--- 1 020-039 7 7 7 11 11 11 T F T <--- 2 040-059 4 4 4 9 9 9 T F T <--- 3 060-079 11 11 11 15 15 15 F F F 4 080-099 14 14 14 21 21 21 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 15 15 15 22 22 22 T F T <--- 7 140-159 15 15 15 22 22 22 F F F 8 160-179 14 14 14 21 21 21 F F F 9 180-199 7 7 7 11 11 11 T T T 10 200-219 10 10 10 14 14 14 T T T 11 220-239 13 13 13 18 18 18 F F F 12 240-259 14 14 14 19 19 19 T F T <--- 13 260-279 12 12 12 17 17 17 F F F 14 280-299 14 14 14 21 21 21 T F T <--- 15 300-319 12 12 12 16 16 16 T F T <--- 16 320-339 10 10 10 14 14 14 T F T <--- 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 14 14 14 21 21 21 F F F 20 400-419 8 8 8 12 12 12 T F T <--- 21 420-439 15 15 15 22 22 22 T F T <--- 22 440-459 14 14 14 20 20 20 T F T <--- 23 460-479 13 13 13 18 18 18 F F F 24 480-499 12 12 12 17 17 17 T F T <--- 25 500-519 3 3 3 8 8 8 T F T <--- 26 520-539 11 11 11 15 15 15 F F F 27 540-559 14 14 14 19 19 19 F F F 28 560-579 14 14 14 19 19 19 F F F 29 580-599 13 13 13 18 18 18 F F F Max gen: 15, Max flush: 22 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 23 23 23 F F F 1 020-039 16 16 16 23 23 23 F F F 2 040-059 16 16 16 23 23 23 F F F 3 060-079 11 11 11 15 15 15 F F F 4 080-099 14 14 14 21 21 21 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 15 15 15 22 22 22 F F F 8 160-179 14 14 14 21 21 21 F F F 9 180-199 16 16 16 23 23 23 F F F 10 200-219 16 16 16 23 23 23 F F F 11 220-239 13 13 13 18 18 18 F F F 12 240-259 16 16 16 23 23 23 F F F 13 260-279 12 12 12 17 17 17 F F F 14 280-299 16 16 16 23 23 23 F F F 15 300-319 16 16 16 23 23 23 F F F 16 320-339 16 16 16 23 23 23 F F F 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 14 14 14 21 21 21 F F F 20 400-419 16 16 16 23 23 23 F F F 21 420-439 16 16 16 23 23 23 F F F 22 440-459 16 16 16 23 23 23 F F F 23 460-479 13 13 13 18 18 18 F F F 24 480-499 16 16 16 23 23 23 F F F 25 500-519 16 16 16 23 23 23 F F F 26 520-539 11 11 11 15 15 15 F F F 27 540-559 14 14 14 19 19 19 F F F 28 560-579 14 14 14 19 19 19 F F F 29 580-599 13 13 13 18 18 18 F F F Max gen: 16, Max flush: 23 Begin loop 9 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.42466809Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:40.4353147Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.435360201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.44576981Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446062106Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446100562Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446112744Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446130483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274} {"msg":"Crucible f7acee45-07b9-416c-9f42-5e505db4f87e has session id: e2e8835c-bcba-4116-adcb-c45b210f8dfb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446464053Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Upstairs opts: Upstairs UUID: f7acee45-07b9-416c-9f42-5e505db4f87e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446490087Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Crucible stats registered with UUID: f7acee45-07b9-416c-9f42-5e505db4f87e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446504798Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:40.446518019Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446683036Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"io task","client":"2","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446721022Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"io task","client":"0","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446736582Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"io task","client":"1","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446830493Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"task":"crutest"} {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446845753Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274} {"msg":"f7acee45-07b9-416c-9f42-5e505db4f87e active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446864122Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446877074Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"client":"0","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446891605Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"client":"1","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446905626Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"client":"2","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446919217Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"io task","client":"2","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446933768Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"io task","client":"0","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.446948249Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"io task","client":"1","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"ds_connection connected from Ok(127.0.0.1:48049)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.447024221Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"io task","client":"0","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"ds_connection connected from Ok(127.0.0.1:37812)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.447052334Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"io task","client":"1","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"ds_connection connected from Ok(127.0.0.1:36063)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.447069223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"io task","client":"2","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.447779467Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"client":"1","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.447820991Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"client":"0","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.447841178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"client":"2","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448131096Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448165274Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"[0]R flush_numbers[0..12]: [23, 23, 23, 15, 21, 22, 23, 22, 21, 23, 23, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448243305Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"[0]R generation[0..12]: [16, 16, 16, 11, 14, 15, 16, 15, 14, 16, 16, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448260095Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448276664Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"[1]R flush_numbers[0..12]: [23, 23, 23, 15, 21, 22, 23, 22, 21, 23, 23, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448292204Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"[1]R generation[0..12]: [16, 16, 16, 11, 14, 15, 16, 15, 14, 16, 16, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448307495Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448323795Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"[2]R flush_numbers[0..12]: [23, 23, 23, 15, 21, 22, 23, 22, 21, 23, 23, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448339255Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"[2]R generation[0..12]: [16, 16, 16, 11, 14, 15, 16, 15, 14, 16, 16, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448354645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448371165Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Max found gen is 17","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448384586Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","Wait for a query_work_queue command to finish before sending IO pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Generation requested: 18 >= found:17","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448413028Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Next flush: 24","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.44842654Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448439942Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"":"downstairs","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448453883Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448466595Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"client":"0","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448480516Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"client":"1","":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448494297Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"client":"2"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"":"downstairs","session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448512416Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448526017Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"f7acee45-07b9-416c-9f42-5e505db4f87e is now active with session: e2e8835c-bcba-4116-adcb-c45b210f8dfb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448541198Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.44855399Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274,"session_id":"e2e8835c-bcba-4116-adcb-c45b210f8dfb"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448567072Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274} {"msg":"Activated sub_volume f7acee45-07b9-416c-9f42-5e505db4f87e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:40.448578564Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 183 len 24576 data: 3 4 4 4 4 4 02/30 Read block 17 len 28672 03/30 Write block 293 len 16384 data: 2 2 2 2 04/30 Read block 194 len 24576 05/30 Read block 330 len 32768 06/30 Read block 538 len 8192 07/30 Write block 158 len 20480 data: 2 2 2 2 2 08/30 Read block 6 len 16384 09/30 Flush 10/30 Write block 33 len 16384 data: 2 2 3 3 11/30 Read block 328 len 32768 12/30 Flush 13/30 Write block 304 len 28672 data: 3 3 3 4 5 4 5 14/30 Read block 66 len 28672 15/30 Write block 320 len 4096 data: 3 16/30 Write block 311 len 24576 data: 5 7 7 6 7 5 17/30 Read block 344 len 24576 18/30 Read block 92 len 36864 19/30 Read block 471 len 28672 20/30 Write block 589 len 20480 data: 4 3 3 2 2 21/30 Write block 69 len 32768 data: 3 3 3 3 3 4 4 3 22/30 Write block 60 len 28672 data: 4 3 3 3 3 3 3 23/30 Read block 470 len 36864 24/30 Read block 54 len 20480 25/30 Write block 441 len 24576 data: 4 4 4 5 5 5 26/30 Read block 358 len 4096 27/30 Read block 66 len 36864 28/30 Read block 330 len 20480 29/30 Write block 213 len 20480 data: 2 2 4 5 4 30/30 Write block 186 len 40960 data: 5 5 5 4 4 4 3 3 3 3 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Read 2 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Read 8 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 5 Sent Done Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 1 1 29 Done 3 29 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1011)) Some(JobId(1011)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:40.461226926Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1274} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 23 23 23 F F F 1 020-039 16 18 18 23 25 25 F F F <--- 2 040-059 16 16 16 23 23 23 F F F 3 060-079 11 11 11 15 15 15 F T T <--- 4 080-099 14 14 14 21 21 21 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 15 18 18 22 24 24 F F F <--- 8 160-179 14 18 18 21 24 24 F F F <--- 9 180-199 16 18 18 23 24 24 T T T <--- 10 200-219 16 16 16 23 23 23 F T T <--- 11 220-239 13 13 13 18 18 18 F F F 12 240-259 16 16 16 23 23 23 F F F 13 260-279 12 12 12 17 17 17 F F F 14 280-299 16 18 18 23 24 24 T F F <--- 15 300-319 16 16 16 23 23 23 F T T <--- 16 320-339 16 16 16 23 23 23 F T T <--- 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 14 14 14 21 21 21 F F F 20 400-419 16 16 16 23 23 23 F F F 21 420-439 16 16 16 23 23 23 F F F 22 440-459 16 16 16 23 23 23 F T T <--- 23 460-479 13 13 13 18 18 18 F F F 24 480-499 16 16 16 23 23 23 F F F 25 500-519 16 16 16 23 23 23 F F F 26 520-539 11 11 11 15 15 15 F F F 27 540-559 14 14 14 19 19 19 F F F 28 560-579 14 14 14 19 19 19 F F F 29 580-599 13 13 13 18 18 18 F T T <--- Max gen: 18, Max flush: 25 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 23 23 23 F F F 1 020-039 18 18 18 25 25 25 F F F 2 040-059 16 16 16 23 23 23 F F F 3 060-079 19 19 19 26 26 26 F F F 4 080-099 14 14 14 21 21 21 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 18 18 18 24 24 24 F F F 8 160-179 18 18 18 24 24 24 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 19 19 19 26 26 26 F F F 11 220-239 13 13 13 18 18 18 F F F 12 240-259 16 16 16 23 23 23 F F F 13 260-279 12 12 12 17 17 17 F F F 14 280-299 18 18 18 24 24 24 F F F 15 300-319 19 19 19 26 26 26 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 7 7 7 11 11 11 F F F 18 360-379 8 8 8 12 12 12 F F F 19 380-399 14 14 14 21 21 21 F F F 20 400-419 16 16 16 23 23 23 F F F 21 420-439 16 16 16 23 23 23 F F F 22 440-459 19 19 19 26 26 26 F F F 23 460-479 13 13 13 18 18 18 F F F 24 480-499 16 16 16 23 23 23 F F F 25 500-519 16 16 16 23 23 23 F F F 26 520-539 11 11 11 15 15 15 F F F 27 540-559 14 14 14 19 19 19 F F F 28 560-579 14 14 14 19 19 19 F F F 29 580-599 19 19 19 26 26 26 F F F Max gen: 19, Max flush: 26 Begin loop 10 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.784416343Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.796237993Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.796539153Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.796578489Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.79659307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.796610848Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282} {"msg":"Crucible e82b5e4a-f447-482b-b987-d5f8999e794d has session id: db5a9440-5b8e-494f-a512-ff81908d689e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.796969913Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Upstairs opts: Upstairs UUID: e82b5e4a-f447-482b-b987-d5f8999e794d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797004621Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Crucible stats registered with UUID: e82b5e4a-f447-482b-b987-d5f8999e794d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.79702235Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:44.797036241Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797179751Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"io task","client":"2","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797213959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"io task","client":"0","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.79722933Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"io task","client":"1","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797334424Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"task":"crutest"} {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797348745Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282} {"msg":"e82b5e4a-f447-482b-b987-d5f8999e794d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797366474Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797379156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"client":"0","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797393087Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"client":"1","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797407048Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"client":"2","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797422718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"io task","client":"2","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797437249Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"io task","client":"0","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.79745148Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"io task","client":"1","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"ds_connection connected from Ok(127.0.0.1:62640)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797485529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"io task","client":"0","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"ds_connection connected from Ok(127.0.0.1:49449)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797501579Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"io task","client":"1","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"ds_connection connected from Ok(127.0.0.1:64955)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.797516759Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"io task","client":"2","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798226663Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"client":"2","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.79826328Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"client":"1","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798280759Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"client":"0","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798571007Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798617817Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"[0]R flush_numbers[0..12]: [23, 25, 23, 26, 21, 22, 23, 24, 24, 26, 26, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798659471Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"[0]R generation[0..12]: [16, 18, 16, 19, 14, 15, 16, 18, 18, 19, 19, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.7986762Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.79869275Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"[1]R flush_numbers[0..12]: [23, 25, 23, 26, 21, 22, 23, 24, 24, 26, 26, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.79870855Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"[1]R generation[0..12]: [16, 18, 16, 19, 14, 15, 16, 18, 18, 19, 19, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798726749Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798743458Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"[2]R flush_numbers[0..12]: [23, 25, 23, 26, 21, 22, 23, 24, 24, 26, 26, 18]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798759128Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"[2]R generation[0..12]: [16, 18, 16, 19, 14, 15, 16, 18, 18, 19, 19, 13]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798774499Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798790938Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Max found gen is 20","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:01:44.79880443Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Generation requested: 20 >= found:20","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798822519Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Next flush: 27","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.79883605Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798849552Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"":"downstairs","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798863673Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798876405Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"client":"0","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798890346Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"client":"1","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798904117Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"client":"2","":"downstairs","session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798921047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798934898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"e82b5e4a-f447-482b-b987-d5f8999e794d is now active with session: db5a9440-5b8e-494f-a512-ff81908d689e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798951737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798964449Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282,"session_id":"db5a9440-5b8e-494f-a512-ff81908d689e"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798977851Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282} {"msg":"Activated sub_volume e82b5e4a-f447-482b-b987-d5f8999e794d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:44.798989404Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 224 len 40960 data: 5 3 2 3 3 2 2 2 2 3 02/30 Flush 03/30 Write block 350 len 12288 data: 3 3 3 04/30 Read block 417 len 32768 05/30 Read block 580 len 20480 06/30 Read block 146 len 16384 07/30 Read block 34 len 20480 08/30 Read block 560 len 32768 09/30 Read block 122 len 28672 10/30 Write block 575 len 32768 data: 3 3 3 3 3 3 3 3 11/30 Write block 95 len 8192 data: 5 5 12/30 Read block 526 len 24576 13/30 Write block 527 len 32768 data: 3 3 3 4 4 5 5 5 14/30 Write block 67 len 8192 data: 2 3 15/30 Write block 267 len 20480 data: 3 3 3 3 2 16/30 Read block 303 len 32768 17/30 Write block 70 len 4096 data: 4 18/30 Write block 249 len 12288 data: 5 5 4 19/30 Read block 33 len 28672 20/30 Write block 78 len 16384 data: 3 3 2 2 21/30 Read block 27 len 36864 22/30 Write block 216 len 40960 data: 6 5 4 4 4 4 5 5 6 4 23/30 Write block 362 len 4096 data: 3 24/30 Read block 104 len 16384 25/30 Read block 329 len 32768 26/30 Read block 523 len 40960 27/30 Read block 328 len 8192 28/30 Read block 594 len 16384 29/30 Read block 412 len 12288 30/30 Read block 356 len 40960 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 3 Done Done Done false 1003 Acked Read 8 Done Sent Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Read 4 Done Sent Done false 1006 Acked Read 5 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Read 7 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Write 2 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Write 3 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 1 28 Done 28 1 27 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:44.811368934Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1282} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 23 23 23 F F F 1 020-039 18 18 18 25 25 25 F F F 2 040-059 16 16 16 23 23 23 F F F 3 060-079 19 19 19 26 26 26 T F T <--- 4 080-099 14 14 14 21 21 21 T F T <--- 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 18 18 18 24 24 24 F F F 8 160-179 18 18 18 24 24 24 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 19 19 19 26 26 26 T F T <--- 11 220-239 20 20 20 27 27 27 T F T <--- 12 240-259 16 16 16 23 23 23 T F T <--- 13 260-279 12 12 12 17 17 17 T F T <--- 14 280-299 18 18 18 24 24 24 F F F 15 300-319 19 19 19 26 26 26 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 7 7 7 11 11 11 T T T 18 360-379 8 8 8 12 12 12 T F T <--- 19 380-399 14 14 14 21 21 21 F F F 20 400-419 16 16 16 23 23 23 F F F 21 420-439 16 16 16 23 23 23 F F F 22 440-459 19 19 19 26 26 26 F F F 23 460-479 13 13 13 18 18 18 F F F 24 480-499 16 16 16 23 23 23 F F F 25 500-519 16 16 16 23 23 23 F F F 26 520-539 11 11 11 15 15 15 T F T <--- 27 540-559 14 14 14 19 19 19 F F F 28 560-579 14 14 14 19 19 19 T F T <--- 29 580-599 19 19 19 26 26 26 T F T <--- Max gen: 20, Max flush: 27 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 23 23 23 F F F 1 020-039 18 18 18 25 25 25 F F F 2 040-059 16 16 16 23 23 23 F F F 3 060-079 21 21 21 28 28 28 F F F 4 080-099 21 21 21 28 28 28 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 18 18 18 24 24 24 F F F 8 160-179 18 18 18 24 24 24 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 21 21 21 28 28 28 F F F 11 220-239 21 21 21 28 28 28 F F F 12 240-259 21 21 21 28 28 28 F F F 13 260-279 21 21 21 28 28 28 F F F 14 280-299 18 18 18 24 24 24 F F F 15 300-319 19 19 19 26 26 26 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 21 21 21 28 28 28 F F F 18 360-379 21 21 21 28 28 28 F F F 19 380-399 14 14 14 21 21 21 F F F 20 400-419 16 16 16 23 23 23 F F F 21 420-439 16 16 16 23 23 23 F F F 22 440-459 19 19 19 26 26 26 F F F 23 460-479 13 13 13 18 18 18 F F F 24 480-499 16 16 16 23 23 23 F F F 25 500-519 16 16 16 23 23 23 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 14 14 14 19 19 19 F F F 28 560-579 21 21 21 28 28 28 F F F 29 580-599 21 21 21 28 28 28 F F F Max gen: 21, Max flush: 28 Begin loop 11 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.126887882Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:49.137565182Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.137610374Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.148122748Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.148414844Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.148452261Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.148464123Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.148481912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290} {"msg":"Crucible 1ef454d4-a349-4c8f-89b2-b4ce73266373 has session id: bb7dfbc7-58c9-474c-bc30-ee153d2ef953","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.148833771Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Upstairs opts: Upstairs UUID: 1ef454d4-a349-4c8f-89b2-b4ce73266373, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.148868009Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Crucible stats registered with UUID: 1ef454d4-a349-4c8f-89b2-b4ce73266373","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.148883949Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:49.148897331Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149045658Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"io task","client":"1","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149080956Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"io task","client":"0","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149096396Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"io task","client":"2","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149196403Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"task":"crutest"} {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149211723Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290} {"msg":"1ef454d4-a349-4c8f-89b2-b4ce73266373 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149229422Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149242194Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"client":"0","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149255896Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"client":"1","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149269667Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"client":"2","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149283338Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"io task","client":"2","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149297779Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"io task","client":"0","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.14931208Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"io task","client":"1","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"ds_connection connected from Ok(127.0.0.1:57012)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149368215Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"io task","client":"1","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"ds_connection connected from Ok(127.0.0.1:44629)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149395838Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"io task","client":"0","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"ds_connection connected from Ok(127.0.0.1:64065)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.149412337Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"io task","client":"2","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150113936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"client":"2","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150143938Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"client":"1","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150160537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"client":"0","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150472371Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150516433Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"[0]R flush_numbers[0..12]: [23, 25, 23, 28, 28, 22, 23, 24, 24, 26, 28, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150570959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"[0]R generation[0..12]: [16, 18, 16, 21, 21, 15, 16, 18, 18, 19, 21, 21]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150588048Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150604538Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"[1]R flush_numbers[0..12]: [23, 25, 23, 28, 28, 22, 23, 24, 24, 26, 28, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150619629Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"[1]R generation[0..12]: [16, 18, 16, 21, 21, 15, 16, 18, 18, 19, 21, 21]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150634959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150650959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"[2]R flush_numbers[0..12]: [23, 25, 23, 28, 28, 22, 23, 24, 24, 26, 28, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150666249Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"[2]R generation[0..12]: [16, 18, 16, 21, 21, 15, 16, 18, 18, 19, 21, 21]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.1506813Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.15069756Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150710751Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150726851Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Next flush: 29","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150740233Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:01:49.150753584Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"":"downstairs","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150773362Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150785784Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"client":"0","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150800015Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"client":"1","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150813797Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"client":"2","":"downstairs","session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150828268Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150841679Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"1ef454d4-a349-4c8f-89b2-b4ce73266373 is now active with session: bb7dfbc7-58c9-474c-bc30-ee153d2ef953","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:01:49.15085656Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150871051Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290,"session_id":"bb7dfbc7-58c9-474c-bc30-ee153d2ef953"} {"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150884032Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290} {"msg":"Activated sub_volume 1ef454d4-a349-4c8f-89b2-b4ce73266373","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:49.150900592Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 153 len 36864 data: 2 2 2 2 2 3 3 3 3 02/30 Read block 198 len 36864 03/30 Read block 313 len 12288 04/30 Flush 05/30 Write block 352 len 28672 data: 4 3 2 2 2 2 2 06/30 Read block 353 len 24576 07/30 Write block 394 len 24576 data: 3 4 4 4 3 3 08/30 Write block 580 len 8192 data: 4 4 09/30 Write block 563 len 36864 data: 4 3 3 3 2 2 2 2 2 10/30 Read block 454 len 4096 11/30 Flush 12/30 Write block 455 len 36864 data: 5 5 5 4 5 4 3 3 2 13/30 Read block 512 len 16384 14/30 Flush 15/30 Write block 388 len 24576 data: 3 3 2 2 2 2 16/30 Read block 59 len 12288 17/30 Read block 76 len 8192 18/30 Read block 199 len 8192 19/30 Read block 64 len 8192 20/30 Read block 251 len 40960 21/30 Read block 223 len 16384 22/30 Read block 190 len 16384 23/30 Write block 412 len 32768 data: 2 3 4 4 4 4 4 4 24/30 Write block 270 len 28672 data: 4 3 2 3 3 3 3 25/30 Flush 26/30 Write block 512 len 16384 data: 2 2 2 4 27/30 Read block 420 len 36864 28/30 Write block 287 len 12288 data: 2 3 3 29/30 Read block 349 len 28672 30/30 Read block 152 len 8192 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 9 Sent Done Done false 1002 Acked Read 3 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Write 6 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 2 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 2 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 1 30 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:49.163436104Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1290} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 23 23 23 F F F 1 020-039 18 18 18 25 25 25 F F F 2 040-059 16 16 16 23 23 23 F F F 3 060-079 21 21 21 28 28 28 F F F 4 080-099 21 21 21 28 28 28 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 18 22 22 24 29 29 T F F <--- 8 160-179 18 22 22 24 29 29 T F F <--- 9 180-199 19 19 19 26 26 26 F F F 10 200-219 21 21 21 28 28 28 F F F 11 220-239 21 21 21 28 28 28 F F F 12 240-259 21 21 21 28 28 28 F F F 13 260-279 21 22 22 28 32 32 F F F <--- 14 280-299 18 18 18 24 24 24 F T T <--- 15 300-319 19 19 19 26 26 26 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 21 22 22 28 30 30 F F F <--- 18 360-379 21 21 21 28 28 28 F F F 19 380-399 14 22 22 21 32 32 F F F <--- 20 400-419 16 22 22 23 32 32 F F F <--- 21 420-439 16 16 16 23 23 23 F F F 22 440-459 19 22 22 26 31 31 F F F <--- 23 460-479 13 22 22 18 31 31 F F F <--- 24 480-499 16 16 16 23 23 23 F F F 25 500-519 16 16 16 23 23 23 F T T <--- 26 520-539 21 21 21 28 28 28 F F F 27 540-559 14 14 14 19 19 19 F F F 28 560-579 21 22 22 28 30 30 F F F <--- 29 580-599 21 22 22 28 30 30 F F F <--- Max gen: 22, Max flush: 32 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 23 23 23 F F F 1 020-039 18 18 18 25 25 25 F F F 2 040-059 16 16 16 23 23 23 F F F 3 060-079 21 21 21 28 28 28 F F F 4 080-099 21 21 21 28 28 28 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 22 22 22 29 29 29 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 21 21 21 28 28 28 F F F 11 220-239 21 21 21 28 28 28 F F F 12 240-259 21 21 21 28 28 28 F F F 13 260-279 22 22 22 32 32 32 F F F 14 280-299 23 23 23 33 33 33 F F F 15 300-319 19 19 19 26 26 26 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 22 22 22 30 30 30 F F F 18 360-379 21 21 21 28 28 28 F F F 19 380-399 22 22 22 32 32 32 F F F 20 400-419 22 22 22 32 32 32 F F F 21 420-439 16 16 16 23 23 23 F F F 22 440-459 22 22 22 31 31 31 F F F 23 460-479 22 22 22 31 31 31 F F F 24 480-499 16 16 16 23 23 23 F F F 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 14 14 14 19 19 19 F F F 28 560-579 22 22 22 30 30 30 F F F 29 580-599 22 22 22 30 30 30 F F F Max gen: 23, Max flush: 33 Begin loop 12 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.485003471Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.496757544Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497071367Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497112701Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497128791Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.49714699Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298} {"msg":"Crucible 5c14415b-f53d-4e78-adf9-9c2860116a54 has session id: 52278f2e-ec5c-4143-9e71-15e845212932","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.49746418Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Upstairs opts: Upstairs UUID: 5c14415b-f53d-4e78-adf9-9c2860116a54, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497498659Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Crucible stats registered with UUID: 5c14415b-f53d-4e78-adf9-9c2860116a54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497514729Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:53.49752819Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497674388Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"io task","client":"1","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497699083Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"io task","client":"2","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497713724Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"io task","client":"0","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497827892Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"task":"crutest"} {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497844082Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298} {"msg":"5c14415b-f53d-4e78-adf9-9c2860116a54 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497861571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497874183Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"client":"0","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497888074Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"client":"1","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497901965Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"client":"2","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497959979Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"io task","client":"2","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.497985683Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"io task","client":"0","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.498001323Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"io task","client":"1","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"ds_connection connected from Ok(127.0.0.1:63558)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.498016683Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"io task","client":"0","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"ds_connection connected from Ok(127.0.0.1:49323)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.498031524Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"io task","client":"1","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"ds_connection connected from Ok(127.0.0.1:58995)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.498139096Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"io task","client":"2","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.498771689Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"client":"0","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.498811394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"client":"2","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.498828363Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"client":"1","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499137199Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.49918367Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"[0]R flush_numbers[0..12]: [23, 25, 23, 28, 28, 22, 23, 29, 29, 26, 28, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499235867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"[0]R generation[0..12]: [16, 18, 16, 21, 21, 15, 16, 22, 22, 19, 21, 21]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499252906Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499269706Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"[1]R flush_numbers[0..12]: [23, 25, 23, 28, 28, 22, 23, 29, 29, 26, 28, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499285206Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"[1]R generation[0..12]: [16, 18, 16, 21, 21, 15, 16, 22, 22, 19, 21, 21]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499303225Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499319734Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"[2]R flush_numbers[0..12]: [23, 25, 23, 28, 28, 22, 23, 29, 29, 26, 28, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499334995Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"[2]R generation[0..12]: [16, 18, 16, 21, 21, 15, 16, 22, 22, 19, 21, 21]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499350085Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499366105Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499380146Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:01:53.499394437Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Next flush: 34","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499412096Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499425678Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"":"downstairs","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499439849Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499452981Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"client":"0","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499467361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"client":"1","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499481433Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"client":"2","":"downstairs","session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499495904Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 crucible","level":30,"time":"2026-03-14T05:01:53.499514482Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"5c14415b-f53d-4e78-adf9-9c2860116a54 is now active with session: 52278f2e-ec5c-4143-9e71-15e845212932","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499530132Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499542784Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298,"session_id":"52278f2e-ec5c-4143-9e71-15e845212932"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499556206Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298} {"msg":"Activated sub_volume 5c14415b-f53d-4e78-adf9-9c2860116a54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:53.499567678Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 540 len 36864 data: 2 2 2 2 2 2 3 4 4 02/30 Read block 60 len 36864 03/30 Write block 307 len 36864 data: 5 6 5 6 6 8 8 7 8 04/30 Read block 469 len 32768 05/30 Write block 299 len 40960 data: 2 2 3 3 4 4 4 4 6 7 06/30 Write block 427 len 40960 data: 3 2 2 2 3 3 3 3 4 4 07/30 Write block 365 len 28672 data: 2 4 4 4 4 2 2 08/30 Write block 577 len 40960 data: 4 4 4 5 5 4 4 4 3 3 09/30 Read block 476 len 24576 10/30 Read block 266 len 36864 11/30 Write block 15 len 28672 data: 3 3 3 3 3 3 3 12/30 Read block 348 len 20480 13/30 Read block 294 len 8192 14/30 Read block 457 len 12288 15/30 Read block 114 len 12288 16/30 Read block 413 len 8192 17/30 Read block 479 len 4096 18/30 Write block 39 len 28672 data: 3 3 3 3 2 2 2 19/30 Read block 594 len 4096 20/30 Read block 71 len 16384 21/30 Read block 221 len 28672 22/30 Write block 491 len 16384 data: 4 3 2 2 23/30 Read block 591 len 4096 24/30 Read block 186 len 24576 25/30 Write block 490 len 4096 data: 4 26/30 Write block 478 len 16384 data: 3 4 5 4 27/30 Read block 340 len 36864 28/30 Read block 154 len 24576 29/30 Write block 59 len 16384 data: 3 5 4 4 30/30 Write block 587 len 4096 data: 3 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 9 Done Sent Done false 1003 Acked Read 8 Done Sent Done false 1004 Acked Write 10 Done Sent Done false 1005 Acked Write 10 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Write 10 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Read 3 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Write 7 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 7 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Read 1 Done Sent Done false 1023 Acked Read 6 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Write 4 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 28 2 32 Done 28 2 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:53.51192904Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1298} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 16 16 16 23 23 23 T F T <--- 1 020-039 18 18 18 25 25 25 T F T <--- 2 040-059 16 16 16 23 23 23 T F T <--- 3 060-079 21 21 21 28 28 28 T F T <--- 4 080-099 21 21 21 28 28 28 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 22 22 22 29 29 29 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 21 21 21 28 28 28 F F F 11 220-239 21 21 21 28 28 28 F F F 12 240-259 21 21 21 28 28 28 F F F 13 260-279 22 22 22 32 32 32 F F F 14 280-299 23 23 23 33 33 33 T F T <--- 15 300-319 19 19 19 26 26 26 T F T <--- 16 320-339 19 19 19 26 26 26 F F F 17 340-359 22 22 22 30 30 30 F F F 18 360-379 21 21 21 28 28 28 T F T <--- 19 380-399 22 22 22 32 32 32 F F F 20 400-419 22 22 22 32 32 32 F F F 21 420-439 16 16 16 23 23 23 T F T <--- 22 440-459 22 22 22 31 31 31 F F F 23 460-479 22 22 22 31 31 31 T F T <--- 24 480-499 16 16 16 23 23 23 T F T <--- 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 14 14 14 19 19 19 T T T 28 560-579 22 22 22 30 30 30 T F T <--- 29 580-599 22 22 22 30 30 30 T F T <--- Max gen: 23, Max flush: 33 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 34 34 34 F F F 1 020-039 24 24 24 34 34 34 F F F 2 040-059 24 24 24 34 34 34 F F F 3 060-079 24 24 24 34 34 34 F F F 4 080-099 21 21 21 28 28 28 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 22 22 22 29 29 29 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 21 21 21 28 28 28 F F F 11 220-239 21 21 21 28 28 28 F F F 12 240-259 21 21 21 28 28 28 F F F 13 260-279 22 22 22 32 32 32 F F F 14 280-299 24 24 24 34 34 34 F F F 15 300-319 24 24 24 34 34 34 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 22 22 22 30 30 30 F F F 18 360-379 24 24 24 34 34 34 F F F 19 380-399 22 22 22 32 32 32 F F F 20 400-419 22 22 22 32 32 32 F F F 21 420-439 24 24 24 34 34 34 F F F 22 440-459 22 22 22 31 31 31 F F F 23 460-479 24 24 24 34 34 34 F F F 24 480-499 24 24 24 34 34 34 F F F 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 24 24 24 34 34 34 F F F 28 560-579 24 24 24 34 34 34 F F F 29 580-599 24 24 24 34 34 34 F F F Max gen: 24, Max flush: 34 Begin loop 13 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.836112393Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:57.846681501Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.846739884Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.857183061Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.857489269Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.857529853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.857541946Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.857559835Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306} {"msg":"Crucible f8883735-74bf-438d-8868-95f339640ffc has session id: 2882670c-d466-4710-a128-2684a099ff4a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.857877555Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Upstairs opts: Upstairs UUID: f8883735-74bf-438d-8868-95f339640ffc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.857911604Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Crucible stats registered with UUID: f8883735-74bf-438d-8868-95f339640ffc","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.857927943Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:57.857944683Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858091561Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"io task","client":"2","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858115156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"io task","client":"1","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858130186Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"io task","client":"0","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858246423Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"task":"crutest"} {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858261074Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306} {"msg":"f8883735-74bf-438d-8868-95f339640ffc active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858277194Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858289796Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"client":"0","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858303907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"client":"1","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858317798Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"client":"2","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.85833138Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"io task","client":"2","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858346131Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"io task","client":"0","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858360731Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"io task","client":"1","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:36843)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858445208Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"io task","client":"2","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:60715)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858527557Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"io task","client":"1","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:49572)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.858554779Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"io task","client":"0","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859157591Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"client":"1","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859193398Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"client":"2","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859298492Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"client":"0","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859536712Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859574629Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"[0]R flush_numbers[0..12]: [34, 34, 34, 34, 28, 22, 23, 29, 29, 26, 28, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859649332Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"[0]R generation[0..12]: [24, 24, 24, 24, 21, 15, 16, 22, 22, 19, 21, 21]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859665941Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859682641Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"[1]R flush_numbers[0..12]: [34, 34, 34, 34, 28, 22, 23, 29, 29, 26, 28, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859697761Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"[1]R generation[0..12]: [24, 24, 24, 24, 21, 15, 16, 22, 22, 19, 21, 21]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859713092Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859729501Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"[2]R flush_numbers[0..12]: [34, 34, 34, 34, 28, 22, 23, 29, 29, 26, 28, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859745021Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"[2]R generation[0..12]: [24, 24, 24, 24, 21, 15, 16, 22, 22, 19, 21, 21]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859759892Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859776282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Max found gen is 25","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859791932Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Generation requested: 26 >= found:25","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859811859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Next flush: 35","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859825251Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859838703Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"":"downstairs","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859852544Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859865366Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"client":"0","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859879557Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"client":"1","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859893278Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "client":"2","":"downstairs","session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859910617Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859923769Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"f8883735-74bf-438d-8868-95f339640ffc is now active with session: 2882670c-d466-4710-a128-2684a099ff4a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.85993853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859951971Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306,"session_id":"2882670c-d466-4710-a128-2684a099ff4a"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859965143Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306} {"msg":"Activated sub_volume f8883735-74bf-438d-8868-95f339640ffc","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:01:57.859976496Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 345 len 12288 data: 2 3 3 02/30 Write block 449 len 32768 data: 4 4 4 3 3 3 6 6 03/30 Flush 04/30 Write block 302 len 32768 data: 4 5 5 5 5 7 8 6 05/30 Write block 540 len 8192 data: 3 3 06/30 Write block 288 len 16384 data: 4 4 3 3 07/30 Write block 407 len 8192 data: 3 3 08/30 Write block 361 len 32768 data: 3 4 2 2 3 5 5 5 09/30 Read block 363 len 40960 10/30 Write block 142 len 8192 data: 3 3 11/30 Flush 12/30 Write block 291 len 12288 data: 4 3 3 13/30 Read block 179 len 4096 14/30 Read block 260 len 8192 15/30 Read block 435 len 36864 16/30 Flush 17/30 Write block 217 len 16384 data: 6 5 5 5 18/30 Read block 506 len 36864 19/30 Read block 189 len 8192 20/30 Read block 40 len 20480 21/30 Write block 369 len 8192 data: 5 3 22/30 Read block 100 len 36864 23/30 Write block 235 len 16384 data: 3 3 3 3 24/30 Read block 256 len 8192 25/30 Read block 586 len 12288 26/30 Read block 558 len 8192 27/30 Write block 48 len 24576 data: 2 2 2 2 2 2 28/30 Read block 129 len 8192 29/30 Write block 553 len 28672 data: 3 4 4 4 4 5 5 30/30 Read block 258 len 8192 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Write 2 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 10 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Read 1 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Write 7 Sent Done Done false 1029 Acked Read 2 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:01:57.871241157Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1306} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 34 34 34 F F F 1 020-039 24 24 24 34 34 34 F F F 2 040-059 24 24 24 34 34 34 F T T <--- 3 060-079 24 24 24 34 34 34 F F F 4 080-099 21 21 21 28 28 28 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 22 26 26 29 36 36 F F F <--- 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 21 21 21 28 28 28 F T T <--- 11 220-239 21 21 21 28 28 28 F T T <--- 12 240-259 21 21 21 28 28 28 F F F 13 260-279 22 22 22 32 32 32 F F F 14 280-299 24 26 26 34 37 37 F F F <--- 15 300-319 24 26 26 34 36 36 F F F <--- 16 320-339 19 19 19 26 26 26 F F F 17 340-359 22 26 26 30 35 35 T F F <--- 18 360-379 24 26 26 34 36 36 F T T <--- 19 380-399 22 22 22 32 32 32 F F F 20 400-419 22 26 26 32 36 36 F F F <--- 21 420-439 24 24 24 34 34 34 F F F 22 440-459 22 26 26 31 35 35 F F F <--- 23 460-479 24 24 24 34 34 34 F F F 24 480-499 24 24 24 34 34 34 F F F 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 24 26 26 34 36 36 F T T <--- 28 560-579 24 24 24 34 34 34 F F F 29 580-599 24 24 24 34 34 34 F F F Max gen: 26, Max flush: 37 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 13 Verifying data now No change after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 34 34 34 F F F 1 020-039 24 24 24 34 34 34 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 24 24 24 34 34 34 F F F 4 080-099 21 21 21 28 28 28 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 26 26 26 36 36 36 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 27 27 27 38 38 38 F F F 11 220-239 27 27 27 38 38 38 F F F 12 240-259 21 21 21 28 28 28 F F F 13 260-279 22 22 22 32 32 32 F F F 14 280-299 26 26 26 37 37 37 F F F 15 300-319 26 26 26 36 36 36 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 26 26 26 35 35 35 F F F 18 360-379 27 27 27 38 38 38 F F F 19 380-399 22 22 22 32 32 32 F F F 20 400-419 26 26 26 36 36 36 F F F 21 420-439 24 24 24 34 34 34 F F F 22 440-459 26 26 26 35 35 35 F F F 23 460-479 24 24 24 34 34 34 F F F 24 480-499 24 24 24 34 34 34 F F F 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 27 27 27 38 38 38 F F F 28 560-579 24 24 24 34 34 34 F F F 29 580-599 24 24 24 34 34 34 F F F Max gen: 27, Max flush: 38 Begin loop 14 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.192310356Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:02.202904309Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.202962223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.213195842Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.213490576Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.213529222Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.213541055Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.213558773Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314} {"msg":"Crucible e1d0c237-69b1-4c59-9cf7-bdfaa9a7c267 has session id: 1a66581a-812f-4be4-a575-7db710e72a04","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.213933818Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Upstairs opts: Upstairs UUID: e1d0c237-69b1-4c59-9cf7-bdfaa9a7c267, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.213968656Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Crucible stats registered with UUID: e1d0c237-69b1-4c59-9cf7-bdfaa9a7c267","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.213984296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:02.213997498Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214154069Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"io task","client":"2","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214178734Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"io task","client":"0","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214193664Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"io task","client":"1","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214313279Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"task":"crutest"} {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214330538Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314} {"msg":"e1d0c237-69b1-4c59-9cf7-bdfaa9a7c267 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214351395Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214366256Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"client":"0","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214380297Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"client":"1","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214394248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"client":"2","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.21440786Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"io task","client":"2","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214422041Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"io task","client":"0","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214436642Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"io task","client":"1","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"ds_connection connected from Ok(127.0.0.1:50533)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214475817Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"io task","client":"1","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"ds_connection connected from Ok(127.0.0.1:36532)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214491887Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"io task","client":"2","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"ds_connection connected from Ok(127.0.0.1:60058)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.214613521Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"io task","client":"0","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.21523583Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"client":"1","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215263272Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"client":"2","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215280491Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"client":"0","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215596283Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215611533Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"[0]R flush_numbers[0..12]: [34, 34, 38, 34, 28, 22, 23, 36, 29, 26, 38, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215644932Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"[0]R generation[0..12]: [24, 24, 27, 24, 21, 15, 16, 26, 22, 19, 27, 27]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.21566431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.21568117Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"[1]R flush_numbers[0..12]: [34, 34, 38, 34, 28, 22, 23, 36, 29, 26, 38, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.21569639Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"[1]R generation[0..12]: [24, 24, 27, 24, 21, 15, 16, 26, 22, 19, 27, 27]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.21571187Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.21572805Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"[2]R flush_numbers[0..12]: [34, 34, 38, 34, 28, 22, 23, 36, 29, 26, 38, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215743501Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"[2]R generation[0..12]: [24, 24, 27, 24, 21, 15, 16, 26, 22, 19, 27, 27]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215759011Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215775171Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Max found gen is 28","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215788652Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"Wait for a query_work_queue command to finish before sending IO } {"msg":"Generation requested: 28 >= found:28","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.2158079Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Next flush: 39","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215821601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215835013Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"":"downstairs","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215849264Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215861966Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"client":"0","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215877216Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"client":"1","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215891488Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"client":"2","":"downstairs","session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:02:02.215905898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215920939Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"e1d0c237-69b1-4c59-9cf7-bdfaa9a7c267 is now active with session: 1a66581a-812f-4be4-a575-7db710e72a04","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.21593602Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215948542Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314,"session_id":"1a66581a-812f-4be4-a575-7db710e72a04"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215961623Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314} {"msg":"Activated sub_volume e1d0c237-69b1-4c59-9cf7-bdfaa9a7c267","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:02.215973066Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 254 len 28672 data: 2 2 3 3 3 3 3 02/30 Write block 215 len 40960 data: 5 7 7 6 6 6 5 6 6 7 03/30 Read block 375 len 16384 04/30 Read block 564 len 16384 05/30 Write block 292 len 36864 data: 4 4 3 3 3 2 2 3 3 06/30 Read block 90 len 20480 07/30 Read block 48 len 32768 08/30 Write block 437 len 8192 data: 4 4 09/30 Write block 88 len 40960 data: 4 4 5 4 5 5 4 6 6 3 10/30 Read block 592 len 4096 11/30 Read block 208 len 24576 12/30 Read block 29 len 40960 13/30 Read block 540 len 4096 14/30 Read block 255 len 32768 15/30 Flush 16/30 Flush 17/30 Write block 212 len 24576 data: 2 3 3 6 8 8 18/30 Read block 70 len 36864 19/30 Read block 279 len 20480 20/30 Write block 61 len 12288 data: 5 5 4 21/30 Write block 72 len 4096 data: 4 22/30 Read block 286 len 20480 23/30 Read block 369 len 32768 24/30 Write block 550 len 8192 data: 3 3 25/30 Read block 299 len 32768 26/30 Write block 299 len 4096 data: 4 27/30 Read block 271 len 16384 28/30 Write block 476 len 20480 data: 2 2 4 5 6 29/30 Read block 277 len 24576 30/30 Read block 560 len 24576 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Read 4 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Read 5 Done Done Done false 1006 Acked Read 8 Done Done Done false 1007 Acked Write 2 Done Done Done false 1008 Acked Write 10 Done Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Read 1 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Write 3 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Read 5 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 5 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Read 6 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 21 0 1 22 Done 9 30 29 68 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:02.228743881Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1314} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 34 34 34 F F F 1 020-039 24 24 24 34 34 34 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 24 24 24 34 34 34 F T T <--- 4 080-099 21 28 28 28 39 39 T F F <--- 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 26 26 26 36 36 36 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 27 28 28 38 39 39 T T T <--- 11 220-239 27 28 28 38 39 39 T F F <--- 12 240-259 21 28 28 28 39 39 T F F <--- 13 260-279 22 28 28 32 39 39 T F F <--- 14 280-299 26 28 28 37 39 39 T T T <--- 15 300-319 26 28 28 36 39 39 T F F <--- 16 320-339 19 19 19 26 26 26 F F F 17 340-359 26 26 26 35 35 35 F F F 18 360-379 27 27 27 38 38 38 F F F 19 380-399 22 22 22 32 32 32 F F F 20 400-419 26 26 26 36 36 36 F F F 21 420-439 24 28 28 34 39 39 T F F <--- 22 440-459 26 26 26 35 35 35 F F F 23 460-479 24 24 24 34 34 34 F T T <--- 24 480-499 24 24 24 34 34 34 F T T <--- 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 27 27 27 38 38 38 F T T <--- 28 560-579 24 24 24 34 34 34 F F F 29 580-599 24 24 24 34 34 34 F F F Max gen: 28, Max flush: 39 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 34 34 34 F F F 1 020-039 24 24 24 34 34 34 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 29 29 29 40 40 40 F F F 4 080-099 28 28 28 39 39 39 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 26 26 26 36 36 36 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 29 29 29 40 40 40 F F F 11 220-239 28 28 28 39 39 39 F F F 12 240-259 28 28 28 39 39 39 F F F 13 260-279 28 28 28 39 39 39 F F F 14 280-299 29 29 29 40 40 40 F F F 15 300-319 28 28 28 39 39 39 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 26 26 26 35 35 35 F F F 18 360-379 27 27 27 38 38 38 F F F 19 380-399 22 22 22 32 32 32 F F F 20 400-419 26 26 26 36 36 36 F F F 21 420-439 28 28 28 39 39 39 F F F 22 440-459 26 26 26 35 35 35 F F F 23 460-479 29 29 29 40 40 40 F F F 24 480-499 29 29 29 40 40 40 F F F 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 29 29 29 40 40 40 F F F 28 560-579 24 24 24 34 34 34 F F F 29 580-599 24 24 24 34 34 34 F F F Max gen: 29, Max flush: 40 Begin loop 15 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.556381083Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.568155853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.568469846Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.568507902Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.568520304Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.568538383Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322} {"msg":"Crucible d143618f-7ff7-43c1-b42a-c9b2eb8d5056 has session id: 4f9f505b-7b7b-4d4f-9bbd-80d94b323113","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.568915446Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Upstairs opts: Upstairs UUID: d143618f-7ff7-43c1-b42a-c9b2eb8d5056, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.568950524Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Crucible stats registered with UUID: d143618f-7ff7-43c1-b42a-c9b2eb8d5056","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.568968613Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:06.568982064Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569134378Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"io task","client":"2","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569168817Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"io task","client":"1","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569184427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"io task","client":"0","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569276069Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"task":"crutest"} {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569292829Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322} {"msg":"d143618f-7ff7-43c1-b42a-c9b2eb8d5056 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569308489Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569321041Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"client":"0","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569335022Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"client":"1","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569349044Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"client":"2","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569389258Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"io task","client":"2","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569411554Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"io task","client":"0","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569426105Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"io task","client":"1","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"ds_connection connected from Ok(127.0.0.1:41407)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569441336Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"io task","client":"1","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"ds_connection connected from Ok(127.0.0.1:53663)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.569563579Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"io task","client":"0","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"ds_connection connected from Ok(127.0.0.1:49722)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.56959303Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"io task","client":"2","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570169488Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"client":"0","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.57019763Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"client":"1","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570251916Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"client":"2","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570575863Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570599947Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"[0]R flush_numbers[0..12]: [34, 34, 38, 40, 39, 22, 23, 36, 29, 26, 40, 39]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570648767Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"[0]R generation[0..12]: [24, 24, 27, 29, 28, 15, 16, 26, 22, 19, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570666146Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570683045Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"[1]R flush_numbers[0..12]: [34, 34, 38, 40, 39, 22, 23, 36, 29, 26, 40, 39]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570698386Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"[1]R generation[0..12]: [24, 24, 27, 29, 28, 15, 16, 26, 22, 19, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570713656Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570730216Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"[2]R flush_numbers[0..12]: [34, 34, 38, 40, 39, 22, 23, 36, 29, 26, 40, 39]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570744936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"[2]R generation[0..12]: [24, 24, 27, 29, 28, 15, 16, 26, 22, 19, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570759927Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570778565Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570792197Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:02:06.570806468Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Next flush: 41","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570825366Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570838837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"":"downstairs","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570853038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.57086592Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"client":"0","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570879961Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"client":"1","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570893973Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"client":"2","":"downstairs","session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570908164Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570924014Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"d143618f-7ff7-43c1-b42a-c9b2eb8d5056 is now active with session: 4f9f505b-7b7b-4d4f-9bbd-80d94b323113","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570938834Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570951596Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322,"session_id":"4f9f505b-7b7b-4d4f-9bbd-80d94b323113"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570964788Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322} {"msg":"Activated sub_volume d143618f-7ff7-43c1-b42a-c9b2eb8d5056","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:06.570976261Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 261 len 24576 data: 3 3 3 3 3 3 02/30 Write block 399 len 32768 data: 4 3 3 3 2 2 2 2 03/30 Write block 372 len 12288 data: 2 2 2 04/30 Read block 480 len 28672 05/30 Read block 382 len 32768 06/30 Write block 91 len 4096 data: 5 07/30 Write block 375 len 28672 data: 2 2 2 2 2 2 2 08/30 Write block 378 len 16384 data: 3 3 3 3 09/30 Read block 64 len 4096 10/30 Read block 67 len 32768 11/30 Read block 84 len 4096 12/30 Read block 62 len 8192 13/30 Write block 79 len 36864 data: 4 3 3 2 3 4 5 5 4 14/30 Read block 316 len 4096 15/30 Read block 440 len 36864 16/30 Read block 243 len 28672 17/30 Read block 313 len 4096 18/30 Write block 379 len 24576 data: 4 4 4 2 2 2 19/30 Read block 301 len 40960 20/30 Read block 243 len 20480 21/30 Read block 219 len 4096 22/30 Write block 270 len 12288 data: 5 4 3 23/30 Read block 291 len 24576 24/30 Read block 197 len 12288 25/30 Read block 366 len 20480 26/30 Read block 224 len 8192 27/30 Write block 582 len 36864 data: 5 5 5 4 4 4 3 5 4 28/30 Write block 83 len 40960 data: 4 5 6 6 5 5 5 6 6 6 29/30 Read block 226 len 28672 30/30 Read block 40 len 36864 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Write 3 Done Done Done false 1003 Acked Read 7 Done Done Done false 1004 Acked Read 8 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Write 7 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Read 1 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Write 9 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Write 3 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Read 5 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Write 10 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 26 27 Done 30 29 4 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:06.582889045Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1322} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 34 34 34 F F F 1 020-039 24 24 24 34 34 34 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 29 29 29 40 40 40 T T F <--- 4 080-099 28 28 28 39 39 39 T T F <--- 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 26 26 26 36 36 36 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 29 29 29 40 40 40 F F F 11 220-239 28 28 28 39 39 39 F F F 12 240-259 28 28 28 39 39 39 F F F 13 260-279 28 28 28 39 39 39 T T T 14 280-299 29 29 29 40 40 40 F F F 15 300-319 28 28 28 39 39 39 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 26 26 26 35 35 35 F F F 18 360-379 27 27 27 38 38 38 T T T 19 380-399 22 22 22 32 32 32 T T T 20 400-419 26 26 26 36 36 36 T T T 21 420-439 28 28 28 39 39 39 F F F 22 440-459 26 26 26 35 35 35 F F F 23 460-479 29 29 29 40 40 40 F F F 24 480-499 29 29 29 40 40 40 F F F 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 29 29 29 40 40 40 F F F 28 560-579 24 24 24 34 34 34 F F F 29 580-599 24 24 24 34 34 34 T T F <--- Max gen: 29, Max flush: 40 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 34 34 34 F F F 1 020-039 24 24 24 34 34 34 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 30 30 30 41 41 41 F F F 4 080-099 30 30 30 41 41 41 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 F F F 7 140-159 26 26 26 36 36 36 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 29 29 29 40 40 40 F F F 11 220-239 28 28 28 39 39 39 F F F 12 240-259 28 28 28 39 39 39 F F F 13 260-279 30 30 30 41 41 41 F F F 14 280-299 29 29 29 40 40 40 F F F 15 300-319 28 28 28 39 39 39 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 26 26 26 35 35 35 F F F 18 360-379 30 30 30 41 41 41 F F F 19 380-399 30 30 30 41 41 41 F F F 20 400-419 30 30 30 41 41 41 F F F 21 420-439 28 28 28 39 39 39 F F F 22 440-459 26 26 26 35 35 35 F F F 23 460-479 29 29 29 40 40 40 F F F 24 480-499 29 29 29 40 40 40 F F F 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 29 29 29 40 40 40 F F F 28 560-579 24 24 24 34 34 34 F F F 29 580-599 30 30 30 41 41 41 F F F Max gen: 30, Max flush: 41 Begin loop 16 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.892738374Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.90434538Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.90464768Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.904685507Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.904698209Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.904716117Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330} {"msg":"Crucible fedbfc9d-ac0e-4269-885f-90cbfdb518f1 has session id: 7c8602dd-1332-489c-9609-3ed3c0158784","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905020446Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Upstairs opts: Upstairs UUID: fedbfc9d-ac0e-4269-885f-90cbfdb518f1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905070285Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Crucible stats registered with UUID: fedbfc9d-ac0e-4269-885f-90cbfdb518f1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905085695Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:10.905098507Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905286989Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"io task","client":"0","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905311543Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"io task","client":"1","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905325984Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"io task","client":"2","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905436505Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"task":"crutest"} {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905451615Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330} {"msg":"fedbfc9d-ac0e-4269-885f-90cbfdb518f1 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905471573Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905484195Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"client":"0","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905500954Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"client":"1","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905515165Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"client":"2","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905528777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"io task","client":"2","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905542998Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"io task","client":"1","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905557239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"io task","client":"0","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"ds_connection connected from Ok(127.0.0.1:60455)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905572529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"io task","client":"2","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"ds_connection connected from Ok(127.0.0.1:55643)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.9055875Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"io task","client":"1","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"ds_connection connected from Ok(127.0.0.1:34552)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.905610555Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"io task","client":"0","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906377563Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"client":"1","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.90641437Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"client":"2","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906431379Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"client":"0","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906762201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906810761Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"[0]R flush_numbers[0..12]: [34, 34, 38, 41, 41, 22, 23, 36, 29, 26, 40, 39]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906854993Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"[0]R generation[0..12]: [24, 24, 27, 30, 30, 15, 16, 26, 22, 19, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906871982Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.90689144Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"[1]R flush_numbers[0..12]: [34, 34, 38, 41, 41, 22, 23, 36, 29, 26, 40, 39]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.9069071Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"[1]R generation[0..12]: [24, 24, 27, 30, 30, 15, 16, 26, 22, 19, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906922701Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.90693893Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"[2]R flush_numbers[0..12]: [34, 34, 38, 41, 41, 22, 23, 36, 29, 26, 40, 39]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906954691Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"[2]R generation[0..12]: [24, 24, 27, 30, 30, 15, 16, 26, 22, 19, 29, 28]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906970071Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906986371Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Max found gen is 31","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.906999932Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Generation requested: 32 >= found:31","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907020089Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Next flush: 42","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907033271Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907046483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"":"downstairs","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907060424Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907073096Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"client":"0","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907087097Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"client":"1","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907100839Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"client":"2","":"downstairs","session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907119117Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907132839Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"fedbfc9d-ac0e-4269-885f-90cbfdb518f1 is now active with session: 7c8602dd-1332-489c-9609-3ed3c0158784","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907148219Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907160871Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330,"session_id":"7c8602dd-1332-489c-9609-3ed3c0158784"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907174123Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330} {"msg":"Activated sub_volume fedbfc9d-ac0e-4269-885f-90cbfdb518f1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:10.907185625Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 442 len 24576 data: 5 5 6 6 6 4 02/30 Write block 342 len 12288 data: 2 2 2 03/30 Read block 586 len 32768 04/30 Read block 444 len 12288 05/30 Write block 263 len 16384 data: 4 4 4 4 06/30 Write block 243 len 24576 data: 4 5 5 5 5 6 07/30 Write block 238 len 8192 data: 4 2 08/30 Flush 09/30 Write block 75 len 36864 data: 5 4 3 4 5 4 4 3 5 10/30 Read block 402 len 20480 11/30 Write block 446 len 8192 data: 7 5 12/30 Read block 340 len 40960 13/30 Read block 154 len 28672 14/30 Write block 30 len 32768 data: 2 2 2 3 3 4 4 3 15/30 Write block 134 len 16384 data: 4 4 3 2 16/30 Write block 580 len 24576 data: 6 6 6 6 6 5 17/30 Write block 245 len 36864 data: 6 6 6 7 6 6 5 2 2 18/30 Read block 405 len 12288 19/30 Read block 14 len 12288 20/30 Write block 423 len 8192 data: 4 4 21/30 Write block 590 len 24576 data: 5 4 3 3 2 2 22/30 Read block 409 len 12288 23/30 Read block 583 len 28672 24/30 Read block 563 len 12288 25/30 Read block 407 len 24576 26/30 Write block 201 len 36864 data: 4 4 4 2 2 2 2 2 2 27/30 Read block 580 len 36864 28/30 Read block 297 len 28672 29/30 Read block 273 len 36864 30/30 Write block 205 len 36864 data: 3 3 3 3 3 2 2 3 4 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Read 3 Done Done Sent false 1004 Acked Write 4 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Write 2 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 9 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Write 2 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Read 3 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 27 29 Done 29 29 3 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) Some(JobId(1007)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:10.919226159Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1330} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 34 34 34 F F F 1 020-039 24 24 24 34 34 34 T T F <--- 2 040-059 27 27 27 38 38 38 F F F 3 060-079 30 30 30 41 41 41 T T F <--- 4 080-099 30 30 30 41 41 41 T T F <--- 5 100-119 15 15 15 22 22 22 F F F 6 120-139 16 16 16 23 23 23 T T F <--- 7 140-159 26 26 26 36 36 36 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 29 29 29 40 40 40 T T F <--- 11 220-239 32 32 28 42 42 39 F F F <--- 12 240-259 32 32 28 42 42 39 T T F <--- 13 260-279 32 32 30 42 42 41 F F F <--- 14 280-299 29 29 29 40 40 40 F F F 15 300-319 28 28 28 39 39 39 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 32 32 26 42 42 35 F F T <--- 18 360-379 30 30 30 41 41 41 F F F 19 380-399 30 30 30 41 41 41 F F F 20 400-419 30 30 30 41 41 41 F F F 21 420-439 28 28 28 39 39 39 T T F <--- 22 440-459 32 32 26 42 42 35 T T T <--- 23 460-479 29 29 29 40 40 40 F F F 24 480-499 29 29 29 40 40 40 F F F 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 29 29 29 40 40 40 F F F 28 560-579 24 24 24 34 34 34 F F F 29 580-599 30 30 30 41 41 41 T T F <--- Max gen: 32, Max flush: 42 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 34 34 34 F F F 1 020-039 33 33 33 43 43 43 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 33 33 33 43 43 43 F F F 4 080-099 33 33 33 43 43 43 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 33 33 33 43 43 43 F F F 7 140-159 26 26 26 36 36 36 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 33 33 33 43 43 43 F F F 11 220-239 32 32 32 42 42 42 F F F 12 240-259 33 33 33 43 43 43 F F F 13 260-279 32 32 32 42 42 42 F F F 14 280-299 29 29 29 40 40 40 F F F 15 300-319 28 28 28 39 39 39 F F F 16 320-339 19 19 19 26 26 26 F F F 17 340-359 32 32 32 42 42 42 F F F 18 360-379 30 30 30 41 41 41 F F F 19 380-399 30 30 30 41 41 41 F F F 20 400-419 30 30 30 41 41 41 F F F 21 420-439 33 33 33 43 43 43 F F F 22 440-459 33 33 33 43 43 43 F F F 23 460-479 29 29 29 40 40 40 F F F 24 480-499 29 29 29 40 40 40 F F F 25 500-519 23 23 23 33 33 33 F F F 26 520-539 21 21 21 28 28 28 F F F 27 540-559 29 29 29 40 40 40 F F F 28 560-579 24 24 24 34 34 34 F F F 29 580-599 33 33 33 43 43 43 F F F Max gen: 33, Max flush: 43 Begin loop 17 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.237229715Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:15.247825007Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.24788264Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.258078073Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.258403279Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.258446842Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.258459144Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.258477003Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338} {"msg":"Crucible b39cd433-fdd4-41b1-b226-030b6a7785fc has session id: c34dd2f0-7c5e-420f-8aaa-fffd82ecf552","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.258851667Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Upstairs opts: Upstairs UUID: b39cd433-fdd4-41b1-b226-030b6a7785fc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.258890283Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Crucible stats registered with UUID: b39cd433-fdd4-41b1-b226-030b6a7785fc","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.258906783Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:15.258920304Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259080853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"io task","client":"2","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259116221Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"io task","client":"0","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259132021Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"io task","client":"1","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259238264Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"task":"crutest"} {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259253904Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338} {"msg":"b39cd433-fdd4-41b1-b226-030b6a7785fc active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259269944Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259282886Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"client":"0","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259296967Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"client":"1","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259311298Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"client":"2","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.25932504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"io task","client":"2","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259339611Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"io task","client":"0","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259353912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"io task","client":"1","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"ds_connection connected from Ok(127.0.0.1:59698)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259520667Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"io task","client":"1","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"ds_connection connected from Ok(127.0.0.1:40369)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259580329Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"io task","client":"2","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"ds_connection connected from Ok(127.0.0.1:45582)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.259598278Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"io task","client":"0","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260128545Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"client":"1","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260164642Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"client":"2","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260239465Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"client":"0","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260497163Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260535889Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"[0]R flush_numbers[0..12]: [34, 43, 38, 43, 43, 22, 43, 36, 29, 26, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.26059808Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"[0]R generation[0..12]: [24, 33, 27, 33, 33, 15, 33, 26, 22, 19, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260615759Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260633188Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"[1]R flush_numbers[0..12]: [34, 43, 38, 43, 43, 22, 43, 36, 29, 26, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260648718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"[1]R generation[0..12]: [24, 33, 27, 33, 33, 15, 33, 26, 22, 19, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260664248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260680508Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"[2]R flush_numbers[0..12]: [34, 43, 38, 43, 43, 22, 43, 36, 29, 26, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260695559Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"[2]R generation[0..12]: [24, 33, 27, 33, 33, 15, 33, 26, 22, 19, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260713547Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260731006Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Wait for a query_work_queue command to finish before sending IO Max found gen is 34","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260750854Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260765285Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Next flush: 44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260778876Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260792198Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"":"downstairs","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260806619Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260819631Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"client":"0","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260833662Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"client":"1","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260850271Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"client":"2","":"downstairs","session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260864223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260877854Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"b39cd433-fdd4-41b1-b226-030b6a7785fc is now active with session: c34dd2f0-7c5e-420f-8aaa-fffd82ecf552","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260893154Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260905956Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338,"session_id":"c34dd2f0-7c5e-420f-8aaa-fffd82ecf552"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260919368Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338} {"msg":"Activated sub_volume b39cd433-fdd4-41b1-b226-030b6a7785fc","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:15.260930701Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 482 len 8192 data: 3 3 02/30 Write block 574 len 20480 data: 2 4 4 5 5 03/30 Write block 1 len 20480 data: 2 2 3 2 2 04/30 Read block 428 len 40960 05/30 Flush 06/30 Write block 124 len 4096 data: 3 07/30 Read block 572 len 16384 08/30 Read block 398 len 40960 09/30 Write block 135 len 28672 data: 5 4 3 3 3 3 3 10/30 Read block 170 len 20480 11/30 Read block 85 len 36864 12/30 Read block 256 len 8192 13/30 Read block 32 len 32768 14/30 Read block 56 len 12288 15/30 Write block 502 len 28672 data: 2 2 2 2 2 2 4 16/30 Write block 532 len 16384 data: 6 6 6 4 17/30 Read block 167 len 20480 18/30 Read block 89 len 32768 19/30 Write block 352 len 8192 data: 5 4 20/30 Read block 323 len 12288 21/30 Flush 22/30 Write block 296 len 28672 data: 4 3 3 5 4 4 5 23/30 Read block 532 len 28672 24/30 Write block 432 len 36864 data: 4 4 4 5 5 5 5 5 5 25/30 Read block 85 len 36864 26/30 Write block 436 len 40960 data: 6 6 6 6 6 5 6 6 7 7 27/30 Read block 495 len 4096 28/30 Write block 326 len 12288 data: 4 4 4 29/30 Write block 344 len 28672 data: 3 3 4 4 3 3 4 30/30 Read block 461 len 4096 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 1 Done Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Write 7 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 4 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Write 7 Sent Done Done false 1029 Acked Read 1 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 0 1 25 Done 1 25 24 50 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:15.273236558Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1338} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 44 44 44 F F F 1 020-039 33 33 33 43 43 43 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 33 33 33 43 43 43 F F F 4 080-099 33 33 33 43 43 43 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 33 34 34 43 45 45 T F F <--- 7 140-159 26 34 34 36 45 45 F F F <--- 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 33 33 33 43 43 43 F F F 11 220-239 32 32 32 42 42 42 F F F 12 240-259 33 33 33 43 43 43 F F F 13 260-279 32 32 32 42 42 42 F F F 14 280-299 29 29 29 40 40 40 F T T <--- 15 300-319 28 28 28 39 39 39 F T T <--- 16 320-339 19 19 19 26 26 26 F T T <--- 17 340-359 32 34 34 42 45 45 F T T <--- 18 360-379 30 30 30 41 41 41 F F F 19 380-399 30 30 30 41 41 41 F F F 20 400-419 30 30 30 41 41 41 F F F 21 420-439 33 33 33 43 43 43 F T T <--- 22 440-459 33 33 33 43 43 43 F T T <--- 23 460-479 29 29 29 40 40 40 F F F 24 480-499 34 34 34 44 44 44 F F F 25 500-519 23 34 34 33 45 45 F F F <--- 26 520-539 21 34 34 28 45 45 F F F <--- 27 540-559 29 29 29 40 40 40 F F F 28 560-579 34 34 34 44 44 44 F F F 29 580-599 33 33 33 43 43 43 F F F Max gen: 34, Max flush: 45 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 44 44 44 F F F 1 020-039 33 33 33 43 43 43 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 33 33 33 43 43 43 F F F 4 080-099 33 33 33 43 43 43 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 34 34 34 45 45 45 F F F 7 140-159 34 34 34 45 45 45 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 33 33 33 43 43 43 F F F 11 220-239 32 32 32 42 42 42 F F F 12 240-259 33 33 33 43 43 43 F F F 13 260-279 32 32 32 42 42 42 F F F 14 280-299 35 35 35 46 46 46 F F F 15 300-319 35 35 35 46 46 46 F F F 16 320-339 35 35 35 46 46 46 F F F 17 340-359 35 35 35 46 46 46 F F F 18 360-379 30 30 30 41 41 41 F F F 19 380-399 30 30 30 41 41 41 F F F 20 400-419 30 30 30 41 41 41 F F F 21 420-439 35 35 35 46 46 46 F F F 22 440-459 35 35 35 46 46 46 F F F 23 460-479 29 29 29 40 40 40 F F F 24 480-499 34 34 34 44 44 44 F F F 25 500-519 34 34 34 45 45 45 F F F 26 520-539 34 34 34 45 45 45 F F F 27 540-559 29 29 29 40 40 40 F F F 28 560-579 34 34 34 44 44 44 F F F 29 580-599 33 33 33 43 43 43 F F F Max gen: 35, Max flush: 46 Begin loop 18 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.593699749Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.605614242Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.605917601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.605956577Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.605968759Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.605986698Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346} {"msg":"Crucible 59cf45f3-774e-4d7e-a4e1-28bcc93422c7 has session id: 1809a53a-a44c-4799-8928-7611f0f8f486","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606328843Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Upstairs opts: Upstairs UUID: 59cf45f3-774e-4d7e-a4e1-28bcc93422c7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606363361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Crucible stats registered with UUID: 59cf45f3-774e-4d7e-a4e1-28bcc93422c7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606379081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:19.606392573Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606544307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"io task","client":"2","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606568572Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"io task","client":"0","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606583183Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"io task","client":"1","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606687377Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"task":"crutest"} {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606702088Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346} {"msg":"59cf45f3-774e-4d7e-a4e1-28bcc93422c7 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606723475Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606738945Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"client":"0","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606753156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"client":"1","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606767377Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"client":"2","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606791402Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"io task","client":"2","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.60681108Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"io task","client":"0","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606825601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"io task","client":"1","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"ds_connection connected from Ok(127.0.0.1:62115)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606840851Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"io task","client":"2","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"ds_connection connected from Ok(127.0.0.1:44043)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.606992686Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"io task","client":"1","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"ds_connection connected from Ok(127.0.0.1:59353)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.607020128Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"io task","client":"0","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.607539392Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"client":"2","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.60757478Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"client":"0","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.607591849Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"client":"1","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.607878629Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.60792439Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"[0]R flush_numbers[0..12]: [44, 43, 38, 43, 43, 22, 45, 45, 29, 26, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.607960977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"[0]R generation[0..12]: [34, 33, 27, 33, 33, 15, 34, 34, 22, 19, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.607978136Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.607997414Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"[1]R flush_numbers[0..12]: [44, 43, 38, 43, 43, 22, 45, 45, 29, 26, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608013244Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"[1]R generation[0..12]: [34, 33, 27, 33, 33, 15, 34, 34, 22, 19, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608028715Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608045414Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"[2]R flush_numbers[0..12]: [44, 43, 38, 43, 43, 22, 45, 45, 29, 26, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608060984Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"[2]R generation[0..12]: [34, 33, 27, 33, 33, 15, 34, 34, 22, 19, 33, 32]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608076505Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608093104Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Max found gen is 36","v":0,"Wait for a query_work_queue command to finish before sending IO name":"crucible","level":30,"time":"2026-03-14T05:02:19.608112292Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608126483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Next flush: 47","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608139865Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608153596Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"":"downstairs","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608167727Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608180519Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"client":"0","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.60819483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"client":"1","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.60821082Z","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"client":"2","":"downstairs","session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608228269Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608241941Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"59cf45f3-774e-4d7e-a4e1-28bcc93422c7 is now active with session: 1809a53a-a44c-4799-8928-7611f0f8f486","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608257081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608270283Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346,"session_id":"1809a53a-a44c-4799-8928-7611f0f8f486"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608283494Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346} {"msg":"Activated sub_volume 59cf45f3-774e-4d7e-a4e1-28bcc93422c7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:19.608294907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 503 len 40960 data: 3 3 3 3 3 5 4 3 2 3 02/30 Read block 165 len 12288 03/30 Flush 04/30 Flush 05/30 Write block 251 len 40960 data: 6 3 3 3 3 4 4 4 4 4 06/30 Write block 239 len 8192 data: 3 3 07/30 Read block 162 len 12288 08/30 Flush 09/30 Write block 569 len 36864 data: 3 3 3 2 2 3 5 5 6 10/30 Read block 293 len 40960 11/30 Write block 342 len 40960 data: 3 3 4 4 5 5 4 4 5 4 12/30 Write block 263 len 20480 data: 5 5 5 5 4 13/30 Read block 348 len 24576 14/30 Read block 148 len 24576 15/30 Write block 512 len 16384 data: 4 3 3 5 16/30 Read block 204 len 20480 17/30 Write block 6 len 24576 data: 2 2 2 2 2 2 18/30 Write block 432 len 16384 data: 5 5 5 6 19/30 Flush 20/30 Write block 576 len 28672 data: 6 7 6 5 7 7 7 21/30 Read block 185 len 8192 22/30 Read block 88 len 36864 23/30 Write block 409 len 24576 data: 3 3 3 3 4 5 24/30 Flush 25/30 Write block 124 len 20480 data: 4 3 3 3 3 26/30 Read block 231 len 28672 27/30 Write block 420 len 32768 data: 3 3 3 5 5 5 4 4 28/30 Read block 375 len 32768 29/30 Read block 465 len 40960 30/30 Read block 265 len 12288 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Sent Done false 1001 Acked Read 3 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 10 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Write 4 Done Sent Done false 1015 Acked Read 5 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Read 8 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:19.621878331Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1346} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 34 36 50 44 50 F F F <--- 1 020-039 33 33 33 43 43 43 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 33 33 33 43 43 43 F F F 4 080-099 33 33 33 43 43 43 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 34 34 34 45 45 45 T F T <--- 7 140-159 34 34 34 45 45 45 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 33 33 33 43 43 43 F F F 11 220-239 36 32 36 49 42 49 F F F <--- 12 240-259 36 33 36 49 43 49 F F F <--- 13 260-279 36 32 36 50 42 50 F F F <--- 14 280-299 35 35 35 46 46 46 F F F 15 300-319 35 35 35 46 46 46 F F F 16 320-339 35 35 35 46 46 46 F F F 17 340-359 36 35 36 50 46 50 F F F <--- 18 360-379 30 30 30 41 41 41 F F F 19 380-399 30 30 30 41 41 41 F F F 20 400-419 36 30 36 51 41 51 F F F <--- 21 420-439 36 35 36 50 46 50 T F T <--- 22 440-459 35 35 35 46 46 46 F F F 23 460-479 29 29 29 40 40 40 F F F 24 480-499 34 34 34 44 44 44 F F F 25 500-519 36 34 36 50 45 50 F F F <--- 26 520-539 34 34 34 45 45 45 F F F 27 540-559 29 29 29 40 40 40 F F F 28 560-579 36 34 36 51 44 51 F F F <--- 29 580-599 36 33 36 51 43 51 F F F <--- Max gen: 36, Max flush: 51 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 50 50 50 F F F 1 020-039 33 33 33 43 43 43 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 33 33 33 43 43 43 F F F 4 080-099 33 33 33 43 43 43 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 37 37 37 52 52 52 F F F 7 140-159 34 34 34 45 45 45 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 33 33 33 43 43 43 F F F 11 220-239 36 36 36 49 49 49 F F F 12 240-259 36 36 36 49 49 49 F F F 13 260-279 36 36 36 50 50 50 F F F 14 280-299 35 35 35 46 46 46 F F F 15 300-319 35 35 35 46 46 46 F F F 16 320-339 35 35 35 46 46 46 F F F 17 340-359 36 36 36 50 50 50 F F F 18 360-379 30 30 30 41 41 41 F F F 19 380-399 30 30 30 41 41 41 F F F 20 400-419 36 36 36 51 51 51 F F F 21 420-439 37 37 37 52 52 52 F F F 22 440-459 35 35 35 46 46 46 F F F 23 460-479 29 29 29 40 40 40 F F F 24 480-499 34 34 34 44 44 44 F F F 25 500-519 36 36 36 50 50 50 F F F 26 520-539 34 34 34 45 45 45 F F F 27 540-559 29 29 29 40 40 40 F F F 28 560-579 36 36 36 51 51 51 F F F 29 580-599 36 36 36 51 51 51 F F F Max gen: 37, Max flush: 52 Begin loop 19 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.935171567Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.946835617Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947155016Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947195271Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947208372Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947226801Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354} {"msg":"Crucible 72a0fa2a-031e-4b07-bcb7-5e21d1489ded has session id: 4dbedec7-6344-411f-879e-b49c074c1838","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947569106Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Upstairs opts: Upstairs UUID: 72a0fa2a-031e-4b07-bcb7-5e21d1489ded, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947602835Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Crucible stats registered with UUID: 72a0fa2a-031e-4b07-bcb7-5e21d1489ded","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947620823Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:23.947634675Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947774667Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"io task","client":"1","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947802979Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"io task","client":"2","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.94781735Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"io task","client":"0","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947921514Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"task":"crutest"} {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947935945Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354} {"msg":"72a0fa2a-031e-4b07-bcb7-5e21d1489ded active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947954903Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947967795Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"client":"0","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947981846Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"client":"1","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.947995997Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"client":"2","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.948010049Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"io task","client":"2","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.9480243Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"io task","client":"0","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.948038921Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"io task","client":"1","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"ds_connection connected from Ok(127.0.0.1:45870)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.948065414Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"io task","client":"0","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"ds_connection connected from Ok(127.0.0.1:37443)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.948080484Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"io task","client":"1","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"ds_connection connected from Ok(127.0.0.1:45296)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.948193673Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"io task","client":"2","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.948777087Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"client":"0","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.948813434Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"client":"1","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.948830643Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"client":"2","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949162055Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949176386Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"[0]R flush_numbers[0..12]: [50, 43, 38, 43, 43, 22, 52, 45, 29, 26, 43, 49]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949241625Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"[0]R generation[0..12]: [36, 33, 27, 33, 33, 15, 37, 34, 22, 19, 33, 36]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949258064Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949274624Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"[1]R flush_numbers[0..12]: [50, 43, 38, 43, 43, 22, 52, 45, 29, 26, 43, 49]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949289964Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"[1]R generation[0..12]: [36, 33, 27, 33, 33, 15, 37, 34, 22, 19, 33, 36]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949305345Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949321734Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"[2]R flush_numbers[0..12]: [50, 43, 38, 43, 43, 22, 52, 45, 29, 26, 43, 49]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949336845Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"[2]R generation[0..12]: [36, 33, 27, 33, 33, 15, 37, 34, 22, 19, 33, 36]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949352035Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949368395Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Max found gen is 38","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949384505Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949398606Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Next flush: 53","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:02:23.949411828Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949428717Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"":"downstairs","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949442828Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.94945542Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"client":"0","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949469851Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"client":"1","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949483483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"client":"2","":"downstairs","session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949497544Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949510596Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"72a0fa2a-031e-4b07-bcb7-5e21d1489ded is now active with session: 4dbedec7-6344-411f-879e-b49c074c1838"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949528125Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949541206Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354,"session_id":"4dbedec7-6344-411f-879e-b49c074c1838"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949554358Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354} {"msg":"Activated sub_volume 72a0fa2a-031e-4b07-bcb7-5e21d1489ded","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:23.949565761Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 233 len 16384 data: 4 3 4 4 02/30 Write block 242 len 28672 data: 4 5 6 7 7 7 8 03/30 Read block 227 len 12288 04/30 Read block 263 len 8192 05/30 Read block 256 len 28672 06/30 Read block 290 len 36864 07/30 Flush 08/30 Write block 268 len 36864 data: 4 4 6 5 4 4 4 4 4 09/30 Read block 483 len 4096 10/30 Flush 11/30 Write block 327 len 24576 data: 5 5 4 5 4 2 12/30 Read block 345 len 4096 13/30 Read block 287 len 32768 14/30 Write block 527 len 8192 data: 4 4 15/30 Flush 16/30 Write block 308 len 20480 data: 9 7 7 7 9 17/30 Read block 156 len 24576 18/30 Read block 56 len 32768 19/30 Write block 208 len 28672 data: 4 4 3 3 4 5 4 20/30 Read block 529 len 28672 21/30 Write block 287 len 24576 data: 3 5 5 4 5 5 22/30 Read block 591 len 12288 23/30 Write block 418 len 16384 data: 5 5 4 4 24/30 Write block 209 len 4096 data: 5 25/30 Write block 302 len 36864 data: 6 6 6 6 6 8 10 8 8 26/30 Read block 293 len 16384 27/30 Write block 5 len 16384 data: 3 3 3 3 28/30 Write block 28 len 16384 data: 2 2 3 3 29/30 Write block 556 len 20480 data: 5 5 6 6 6 30/30 Read block 239 len 16384 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Sent false 1001 Acked Write 7 Done Done Sent false 1002 Acked Read 3 Done Done Sent false 1003 Acked Read 2 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Read 9 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Read 1 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Write 4 Done Done Sent false 1023 Acked Write 1 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Write 5 Done Done Sent false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 30 31 Done 29 30 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1014)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:23.961347067Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1354} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 50 50 50 T T F <--- 1 020-039 33 33 33 43 43 43 T T F <--- 2 040-059 27 27 27 38 38 38 F F F 3 060-079 33 33 33 43 43 43 F F F 4 080-099 33 33 33 43 43 43 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 37 37 37 52 52 52 F F F 7 140-159 34 34 34 45 45 45 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 33 33 33 43 43 43 T T F <--- 11 220-239 38 38 36 53 53 49 F F F <--- 12 240-259 38 38 36 53 53 49 F F F <--- 13 260-279 38 38 36 54 54 50 F F F <--- 14 280-299 35 35 35 46 46 46 T T F <--- 15 300-319 35 35 35 46 46 46 T T F <--- 16 320-339 38 38 35 55 55 46 F F F <--- 17 340-359 36 36 36 50 50 50 F F F 18 360-379 30 30 30 41 41 41 F F F 19 380-399 30 30 30 41 41 41 F F F 20 400-419 36 36 36 51 51 51 T T F <--- 21 420-439 37 37 37 52 52 52 T T F <--- 22 440-459 35 35 35 46 46 46 F F F 23 460-479 29 29 29 40 40 40 F F F 24 480-499 34 34 34 44 44 44 F F F 25 500-519 36 36 36 50 50 50 F F F 26 520-539 38 38 34 55 55 45 F F F <--- 27 540-559 29 29 29 40 40 40 T T F <--- 28 560-579 36 36 36 51 51 51 T T F <--- 29 580-599 36 36 36 51 51 51 F F F Max gen: 38, Max flush: 55 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 56 56 56 F F F 1 020-039 39 39 39 56 56 56 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 33 33 33 43 43 43 F F F 4 080-099 33 33 33 43 43 43 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 37 37 37 52 52 52 F F F 7 140-159 34 34 34 45 45 45 F F F 8 160-179 22 22 22 29 29 29 F F F 9 180-199 19 19 19 26 26 26 F F F 10 200-219 39 39 39 56 56 56 F F F 11 220-239 38 38 38 53 53 53 F F F 12 240-259 38 38 38 53 53 53 F F F 13 260-279 38 38 38 54 54 54 F F F 14 280-299 39 39 39 56 56 56 F F F 15 300-319 39 39 39 56 56 56 F F F 16 320-339 38 38 38 55 55 55 F F F 17 340-359 36 36 36 50 50 50 F F F 18 360-379 30 30 30 41 41 41 F F F 19 380-399 30 30 30 41 41 41 F F F 20 400-419 39 39 39 56 56 56 F F F 21 420-439 39 39 39 56 56 56 F F F 22 440-459 35 35 35 46 46 46 F F F 23 460-479 29 29 29 40 40 40 F F F 24 480-499 34 34 34 44 44 44 F F F 25 500-519 36 36 36 50 50 50 F F F 26 520-539 38 38 38 55 55 55 F F F 27 540-559 39 39 39 56 56 56 F F F 28 560-579 39 39 39 56 56 56 F F F 29 580-599 36 36 36 51 51 51 F F F Max gen: 39, Max flush: 56 Begin loop 20 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.284403539Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.295936481Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.296246556Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.296295206Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.296307948Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.296325917Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362} {"msg":"Crucible b4dd6db4-d38f-44c3-870e-9f4ef6750594 has session id: c2c983d5-4b8d-44a8-bf03-02ba08dce35e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.296619012Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Upstairs opts: Upstairs UUID: b4dd6db4-d38f-44c3-870e-9f4ef6750594, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.29665397Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Crucible stats registered with UUID: b4dd6db4-d38f-44c3-870e-9f4ef6750594","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.29667022Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:28.296683702Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.296824413Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"io task","client":"2","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.296848468Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"io task","client":"0","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.296863279Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"io task","client":"1","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.296986761Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"task":"crutest"} {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297010097Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362} {"msg":"b4dd6db4-d38f-44c3-870e-9f4ef6750594 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297026167Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297038779Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"client":"0","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297055068Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"client":"1","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.29706897Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"client":"2","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297110424Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"io task","client":"2","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297125404Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"io task","client":"0","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297139925Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"io task","client":"1","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"ds_connection connected from Ok(127.0.0.1:34371)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297168687Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"io task","client":"2","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"ds_connection connected from Ok(127.0.0.1:54282)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297183937Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"io task","client":"0","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"ds_connection connected from Ok(127.0.0.1:64877)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297199757Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"io task","client":"1","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297893571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"client":"1","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297930938Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"client":"0","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.297948147Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"client":"2","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298265088Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298311169Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"[0]R flush_numbers[0..12]: [56, 56, 38, 43, 43, 22, 52, 45, 29, 26, 56, 53]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298348026Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"[0]R generation[0..12]: [39, 39, 27, 33, 33, 15, 37, 34, 22, 19, 39, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298364895Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298384673Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"[1]R flush_numbers[0..12]: [56, 56, 38, 43, 43, 22, 52, 45, 29, 26, 56, 53]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298400033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"[1]R generation[0..12]: [39, 39, 27, 33, 33, 15, 37, 34, 22, 19, 39, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298415374Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298431463Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"[2]R flush_numbers[0..12]: [56, 56, 38, 43, 43, 22, 52, 45, 29, 26, 56, 53]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298446834Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"[2]R generation[0..12]: [39, 39, 27, 33, 33, 15, 37, 34, 22, 19, 39, 38]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298462034Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298478164Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298491606Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298505487Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Next flush: 57","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298523556Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298537057Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"":"downstairs","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298551318Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.29856403Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"client":"0","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298578011Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"client":"1","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298592252Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"client":"2","":"downstairs","session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298607793Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:02:28.298621224Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"b4dd6db4-d38f-44c3-870e-9f4ef6750594 is now active with session: c2c983d5-4b8d-44a8-bf03-02ba08dce35e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298639143Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298651745Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362,"session_id":"c2c983d5-4b8d-44a8-bf03-02ba08dce35e"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298664907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362} {"msg":"Activated sub_volume b4dd6db4-d38f-44c3-870e-9f4ef6750594","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:28.298676849Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 491 len 20480 data: 5 4 3 3 2 02/30 Read block 100 len 40960 03/30 Flush 04/30 Write block 182 len 16384 data: 3 4 5 5 05/30 Read block 568 len 16384 06/30 Read block 480 len 4096 07/30 Read block 194 len 28672 08/30 Read block 457 len 28672 09/30 Write block 74 len 24576 data: 5 6 5 4 5 6 10/30 Write block 509 len 16384 data: 5 4 3 5 11/30 Read block 368 len 4096 12/30 Write block 393 len 36864 data: 3 4 5 5 5 4 5 4 4 13/30 Read block 409 len 36864 14/30 Write block 386 len 20480 data: 3 3 4 4 3 15/30 Write block 176 len 16384 data: 2 2 2 2 16/30 Write block 284 len 16384 data: 4 4 3 4 17/30 Read block 528 len 16384 18/30 Write block 213 len 20480 data: 6 5 7 9 9 19/30 Flush 20/30 Write block 225 len 16384 data: 5 3 4 4 21/30 Read block 347 len 20480 22/30 Write block 367 len 4096 data: 6 23/30 Read block 155 len 20480 24/30 Read block 78 len 32768 25/30 Write block 529 len 16384 data: 4 5 5 7 26/30 Read block 484 len 4096 27/30 Write block 476 len 28672 data: 3 3 5 6 7 5 4 28/30 Read block 468 len 8192 29/30 Read block 142 len 24576 30/30 Read block 251 len 40960 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 4 Done Done Done false 1004 Acked Read 4 Done Done Done false 1005 Acked Read 1 Done Done Done false 1006 Acked Read 7 Done Done Done false 1007 Acked Read 7 Done Done Done false 1008 Acked Write 6 Done Done Sent false 1009 Acked Write 4 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Write 5 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Read 4 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Read 5 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Read 10 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 22 23 Done 26 27 5 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) Some(JobId(1002)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:28.31081733Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1362} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 56 56 56 F F F 1 020-039 39 39 39 56 56 56 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 40 40 33 58 58 43 F F F <--- 4 080-099 33 33 33 43 43 43 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 37 37 37 52 52 52 F F F 7 140-159 34 34 34 45 45 45 F F F 8 160-179 40 40 22 58 58 29 F F F <--- 9 180-199 40 40 19 58 58 26 F F T <--- 10 200-219 40 40 39 58 58 56 F F F <--- 11 220-239 38 38 38 53 53 53 T T F <--- 12 240-259 38 38 38 53 53 53 F F F 13 260-279 38 38 38 54 54 54 F F F 14 280-299 40 40 39 58 58 56 F F F <--- 15 300-319 39 39 39 56 56 56 F F F 16 320-339 38 38 38 55 55 55 F F F 17 340-359 36 36 36 50 50 50 F F F 18 360-379 30 30 30 41 41 41 T T F <--- 19 380-399 40 40 30 58 58 41 F F F <--- 20 400-419 40 40 39 58 58 56 F F F <--- 21 420-439 39 39 39 56 56 56 F F F 22 440-459 35 35 35 46 46 46 F F F 23 460-479 29 29 29 40 40 40 T T F <--- 24 480-499 40 40 40 57 57 57 T T F <--- 25 500-519 40 40 36 58 58 50 F F F <--- 26 520-539 38 38 38 55 55 55 T T F <--- 27 540-559 39 39 39 56 56 56 F F F 28 560-579 39 39 39 56 56 56 F F F 29 580-599 36 36 36 51 51 51 F F F Max gen: 40, Max flush: 58 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 39 39 39 56 56 56 F F F 1 020-039 39 39 39 56 56 56 F F F 2 040-059 27 27 27 38 38 38 F F F 3 060-079 40 40 40 58 58 58 F F F 4 080-099 33 33 33 43 43 43 F F F 5 100-119 15 15 15 22 22 22 F F F 6 120-139 37 37 37 52 52 52 F F F 7 140-159 34 34 34 45 45 45 F F F 8 160-179 40 40 40 58 58 58 F F F 9 180-199 40 40 40 58 58 58 F F F 10 200-219 40 40 40 58 58 58 F F F 11 220-239 41 41 41 59 59 59 F F F 12 240-259 38 38 38 53 53 53 F F F 13 260-279 38 38 38 54 54 54 F F F 14 280-299 40 40 40 58 58 58 F F F 15 300-319 39 39 39 56 56 56 F F F 16 320-339 38 38 38 55 55 55 F F F 17 340-359 36 36 36 50 50 50 F F F 18 360-379 41 41 41 59 59 59 F F F 19 380-399 40 40 40 58 58 58 F F F 20 400-419 40 40 40 58 58 58 F F F 21 420-439 39 39 39 56 56 56 F F F 22 440-459 35 35 35 46 46 46 F F F 23 460-479 41 41 41 59 59 59 F F F 24 480-499 41 41 41 59 59 59 F F F 25 500-519 40 40 40 58 58 58 F F F 26 520-539 41 41 41 59 59 59 F F F 27 540-559 39 39 39 56 56 56 F F F 28 560-579 39 39 39 56 56 56 F F F 29 580-599 36 36 36 51 51 51 F F F Max gen: 41, Max flush: 59 Begin loop 21 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.628132848Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.639783206Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640094661Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640131667Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640144039Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640162018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370} {"msg":"Crucible 77cc58ce-0a48-4943-b294-d7e7bd449c0a has session id: 5d325400-765a-4bdf-8faa-110ce1c773ff","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.64047818Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Upstairs opts: Upstairs UUID: 77cc58ce-0a48-4943-b294-d7e7bd449c0a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640512868Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Crucible stats registered with UUID: 77cc58ce-0a48-4943-b294-d7e7bd449c0a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640528998Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:32.640542459Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640692425Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"io task","client":"1","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640727853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"io task","client":"0","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640744212Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"io task","client":"2","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640856152Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"task":"crutest"} {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640870623Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370} {"msg":"77cc58ce-0a48-4943-b294-d7e7bd449c0a active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640892269Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640906071Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"client":"0","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640921081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"client":"1","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640935212Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"client":"2","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640949044Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"io task","client":"2","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640963315Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"io task","client":"0","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640977636Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"io task","client":"1","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:47181)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.640998443Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"io task","client":"1","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:64227)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.641013773Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"io task","client":"0","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:42168)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.641028544Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"io task","client":"2","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.641689658Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"client":"0","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.641742055Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"client":"1","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.641889233Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"client":"2","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642049482Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642098112Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"[0]R flush_numbers[0..12]: [56, 56, 38, 58, 43, 22, 52, 45, 58, 58, 58, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642243Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"[0]R generation[0..12]: [39, 39, 27, 40, 33, 15, 37, 34, 40, 40, 40, 41]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642260469Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642277209Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"[1]R flush_numbers[0..12]: [56, 56, 38, 58, 43, 22, 52, 45, 58, 58, 58, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642292629Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"[1]R generation[0..12]: [39, 39, 27, 40, 33, 15, 37, 34, 40, 40, 40, 41]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.64230799Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642324799Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"[2]R flush_numbers[0..12]: [56, 56, 38, 58, 43, 22, 52, 45, 58, 58, 58, 59]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642340329Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"[2]R generation[0..12]: [39, 39, 27, 40, 33, 15, 37, 34, 40, 40, 40, 41]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642356369Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642373189Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:02:32.64238684Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642408726Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Next flush: 60","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642422068Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.64243543Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"":"downstairs","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642449631Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642462363Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"client":"0","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642476754Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"client":"1","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Transition from WaitQuorum to Active","v"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642494352Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"client":"2","":"downstairs","session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642509733Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642523244Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"77cc58ce-0a48-4943-b294-d7e7bd449c0a is now active with session: 5d325400-765a-4bdf-8faa-110ce1c773ff","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642538235Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642550717Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370,"session_id":"5d325400-765a-4bdf-8faa-110ce1c773ff"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642563829Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370} {"msg":"Activated sub_volume 77cc58ce-0a48-4943-b294-d7e7bd449c0a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:32.642575452Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 73 len 12288 data: 4 6 7 02/30 Write block 18 len 28672 data: 4 4 4 4 4 2 2 03/30 Read block 23 len 12288 04/30 Write block 291 len 24576 data: 6 6 5 4 4 5 05/30 Read block 371 len 32768 06/30 Flush 07/30 Write block 144 len 4096 data: 3 08/30 Write block 130 len 20480 data: 3 4 4 4 5 09/30 Write block 99 len 36864 data: 2 2 3 4 4 4 4 4 3 10/30 Read block 254 len 40960 11/30 Read block 30 len 8192 12/30 Read block 8 len 40960 13/30 Read block 3 len 4096 14/30 Write block 212 len 8192 data: 5 7 15/30 Write block 235 len 4096 data: 5 16/30 Flush 17/30 Write block 197 len 8192 data: 3 3 18/30 Flush 19/30 Flush 20/30 Write block 151 len 36864 data: 2 2 3 3 3 3 3 4 4 21/30 Write block 477 len 36864 data: 4 6 7 8 6 5 4 3 2 22/30 Flush 23/30 Write block 552 len 36864 data: 3 4 5 5 6 6 7 7 7 24/30 Write block 172 len 8192 data: 3 2 25/30 Read block 296 len 28672 26/30 Write block 52 len 16384 data: 3 3 2 3 27/30 Read block 586 len 16384 28/30 Write block 203 len 36864 data: 5 3 4 4 4 5 6 4 4 29/30 Read block 527 len 24576 30/30 Read block 148 len 24576 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 7 Done Done Sent false 1002 Acked Read 3 Done Done Sent false 1003 Acked Write 6 Done Done Sent false 1004 Acked Read 8 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Write 9 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 9 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Read 7 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:32.654609099Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1370} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 39 60 60 56 F F F <--- 1 020-039 42 42 39 60 60 56 F F F <--- 2 040-059 27 27 27 38 38 38 T T F <--- 3 060-079 42 42 40 60 60 58 F F T <--- 4 080-099 42 42 33 61 61 43 F F F <--- 5 100-119 42 42 15 61 61 22 F F F <--- 6 120-139 42 42 37 61 61 52 F F F <--- 7 140-159 42 42 34 64 64 45 F F F <--- 8 160-179 40 40 40 58 58 58 T T F <--- 9 180-199 42 42 40 62 62 58 F F F <--- 10 200-219 42 42 40 61 61 58 T T F <--- 11 220-239 42 42 41 61 61 59 F F F <--- 12 240-259 38 38 38 53 53 53 F F F 13 260-279 38 38 38 54 54 54 F F F 14 280-299 42 42 40 60 60 58 F F F <--- 15 300-319 39 39 39 56 56 56 F F F 16 320-339 38 38 38 55 55 55 F F F 17 340-359 36 36 36 50 50 50 F F F 18 360-379 41 41 41 59 59 59 F F F 19 380-399 40 40 40 58 58 58 F F F 20 400-419 40 40 40 58 58 58 F F F 21 420-439 39 39 39 56 56 56 F F F 22 440-459 35 35 35 46 46 46 F F F 23 460-479 42 42 41 64 64 59 F F F <--- 24 480-499 42 42 41 64 64 59 F F F <--- 25 500-519 40 40 40 58 58 58 F F F 26 520-539 41 41 41 59 59 59 F F F 27 540-559 39 39 39 56 56 56 T T F <--- 28 560-579 39 39 39 56 56 56 T T F <--- 29 580-599 36 36 36 51 51 51 F F F Max gen: 42, Max flush: 64 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 42 42 42 60 60 60 F F F 1 020-039 42 42 42 60 60 60 F F F 2 040-059 43 43 43 65 65 65 F F F 3 060-079 42 42 42 60 60 60 F F F 4 080-099 42 42 42 61 61 61 F F F 5 100-119 42 42 42 61 61 61 F F F 6 120-139 42 42 42 61 61 61 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 43 43 43 65 65 65 F F F 9 180-199 42 42 42 62 62 62 F F F 10 200-219 43 43 43 65 65 65 F F F 11 220-239 42 42 42 61 61 61 F F F 12 240-259 38 38 38 53 53 53 F F F 13 260-279 38 38 38 54 54 54 F F F 14 280-299 42 42 42 60 60 60 F F F 15 300-319 39 39 39 56 56 56 F F F 16 320-339 38 38 38 55 55 55 F F F 17 340-359 36 36 36 50 50 50 F F F 18 360-379 41 41 41 59 59 59 F F F 19 380-399 40 40 40 58 58 58 F F F 20 400-419 40 40 40 58 58 58 F F F 21 420-439 39 39 39 56 56 56 F F F 22 440-459 35 35 35 46 46 46 F F F 23 460-479 42 42 42 64 64 64 F F F 24 480-499 42 42 42 64 64 64 F F F 25 500-519 40 40 40 58 58 58 F F F 26 520-539 41 41 41 59 59 59 F F F 27 540-559 43 43 43 65 65 65 F F F 28 560-579 43 43 43 65 65 65 F F F 29 580-599 36 36 36 51 51 51 F F F Max gen: 43, Max flush: 65 Begin loop 22 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.982859936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.994520518Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.994841296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.994881501Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.994897641Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.99491555Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378} {"msg":"Crucible 0fad08f5-0a66-414c-abc1-d89d87f111f5 has session id: 98c415b4-35a1-43a0-9ce0-0f8af2095197","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995256215Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Upstairs opts: Upstairs UUID: 0fad08f5-0a66-414c-abc1-d89d87f111f5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995290064Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Crucible stats registered with UUID: 0fad08f5-0a66-414c-abc1-d89d87f111f5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995306124Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:36.995319426Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995492887Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"io task","client":"1","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995526885Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"io task","client":"0","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995542426Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"io task","client":"2","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995639864Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"task":"crutest"} {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995656674Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378} {"msg":"0fad08f5-0a66-414c-abc1-d89d87f111f5 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995673863Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995686675Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"client":"0","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995700666Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"client":"1","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995717466Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"client":"2","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995759759Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"io task","client":"2","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995784783Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"io task","client":"0","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995800224Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"io task","client":"1","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"ds_connection connected from Ok(127.0.0.1:39049)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995815434Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"io task","client":"2","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"ds_connection connected from Ok(127.0.0.1:52284)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.99594841Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"io task","client":"0","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"ds_connection connected from Ok(127.0.0.1:65276)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.995975503Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"io task","client":"1","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.996596863Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"client":"0","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.996625615Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"client":"1","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.996642494Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"client":"2","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.996992165Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997040234Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"[0]R flush_numbers[0..12]: [60, 60, 65, 60, 61, 61, 61, 64, 65, 62, 65, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997107692Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"[0]R generation[0..12]: [42, 42, 43, 42, 42, 42, 42, 42, 43, 42, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997124831Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997141361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"[1]R flush_numbers[0..12]: [60, 60, 65, 60, 61, 61, 61, 64, 65, 62, 65, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997159599Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"[1]R generation[0..12]: [42, 42, 43, 42, 42, 42, 42, 42, 43, 42, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.99717499Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997191559Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"[2]R flush_numbers[0..12]: [60, 60, 65, 60, 61, 61, 61, 64, 65, 62, 65, 61]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.99720709Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"[2]R generation[0..12]: [42, 42, 43, 42, 42, 42, 42, 42, 43, 42, 43, 42]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.99722218Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.99723861Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Max found gen is 44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997252341Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997272988Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Next flush: 66","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.99728609Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997299761Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"":"downstairs","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997313902Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997326345Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"client":"0","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997340646Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"client":"1","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997354437Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"client":"2","":"downstairs","session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997368568Z","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997387116Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"0fad08f5-0a66-414c-abc1-d89d87f111f5 is now active with session: 98c415b4-35a1-43a0-9ce0-0f8af2095197","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997401737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997414209Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378,"session_id":"98c415b4-35a1-43a0-9ce0-0f8af2095197"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997427571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378} {"msg":"Activated sub_volume 0fad08f5-0a66-414c-abc1-d89d87f111f5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:36.997439124Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 232 len 40960 data: 3 5 4 6 5 4 5 4 4 3 03/30 Flush 04/30 Flush 05/30 Write block 205 len 40960 data: 5 5 5 6 7 5 5 6 8 6 06/30 Read block 303 len 40960 07/30 Write block 204 len 8192 data: 4 6 08/30 Write block 183 len 28672 data: 5 6 6 6 6 6 5 09/30 Read block 88 len 24576 10/30 Read block 162 len 24576 11/30 Read block 3 len 12288 12/30 Write block 63 len 20480 data: 5 4 4 4 3 13/30 Read block 225 len 4096 14/30 Write block 0 len 24576 data: 2 3 3 4 3 4 15/30 Flush 16/30 Write block 372 len 36864 data: 3 3 3 3 3 3 4 5 5 17/30 Read block 160 len 28672 18/30 Write block 548 len 20480 data: 5 4 4 4 4 19/30 Read block 480 len 32768 20/30 Write block 331 len 16384 data: 5 3 2 2 21/30 Flush 22/30 Write block 71 len 16384 data: 4 5 5 7 23/30 Read block 222 len 16384 24/30 Read block 103 len 36864 25/30 Read block 213 len 40960 26/30 Read block 394 len 20480 27/30 Write block 557 len 16384 data: 7 8 8 8 28/30 Read block 318 len 32768 29/30 Read block 403 len 40960 30/30 Write block 443 len 4096 data: 7 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 10 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 10 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Read 6 Done Sent Done false 1010 Acked Read 3 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Read 7 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Read 9 Done Sent Done false 1024 Acked Read 10 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Read 8 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1000)) Some(JobId(1020)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:37.01146452Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1378} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 42 44 69 60 69 F F F <--- 1 020-039 42 42 42 60 60 60 F F F 2 040-059 43 43 43 65 65 65 F F F 3 060-079 44 42 44 69 60 69 T F T <--- 4 080-099 42 42 42 61 61 61 F F F 5 100-119 42 42 42 61 61 61 F F F 6 120-139 42 42 42 61 61 61 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 43 43 43 65 65 65 F F F 9 180-199 44 42 44 69 62 69 F F F <--- 10 200-219 44 43 44 69 65 69 F F F <--- 11 220-239 44 42 44 67 61 67 F F F <--- 12 240-259 44 38 44 67 53 67 F F F <--- 13 260-279 38 38 38 54 54 54 F F F 14 280-299 42 42 42 60 60 60 F F F 15 300-319 39 39 39 56 56 56 F F F 16 320-339 44 38 44 70 55 70 F F F <--- 17 340-359 36 36 36 50 50 50 F F F 18 360-379 44 41 44 70 59 70 F F F <--- 19 380-399 44 40 44 70 58 70 F F F <--- 20 400-419 40 40 40 58 58 58 F F F 21 420-439 39 39 39 56 56 56 F F F 22 440-459 35 35 35 46 46 46 T F T <--- 23 460-479 42 42 42 64 64 64 F F F 24 480-499 42 42 42 64 64 64 F F F 25 500-519 40 40 40 58 58 58 F F F 26 520-539 41 41 41 59 59 59 F F F 27 540-559 44 43 44 70 65 70 T F T <--- 28 560-579 43 43 43 65 65 65 T F T <--- 29 580-599 36 36 36 51 51 51 F F F Max gen: 44, Max flush: 70 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 22 Verifying data now No change after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 69 69 69 F F F 1 020-039 42 42 42 60 60 60 F F F 2 040-059 43 43 43 65 65 65 F F F 3 060-079 45 45 45 71 71 71 F F F 4 080-099 42 42 42 61 61 61 F F F 5 100-119 42 42 42 61 61 61 F F F 6 120-139 42 42 42 61 61 61 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 43 43 43 65 65 65 F F F 9 180-199 44 44 44 69 69 69 F F F 10 200-219 44 44 44 69 69 69 F F F 11 220-239 44 44 44 67 67 67 F F F 12 240-259 44 44 44 67 67 67 F F F 13 260-279 38 38 38 54 54 54 F F F 14 280-299 42 42 42 60 60 60 F F F 15 300-319 39 39 39 56 56 56 F F F 16 320-339 44 44 44 70 70 70 F F F 17 340-359 36 36 36 50 50 50 F F F 18 360-379 44 44 44 70 70 70 F F F 19 380-399 44 44 44 70 70 70 F F F 20 400-419 40 40 40 58 58 58 F F F 21 420-439 39 39 39 56 56 56 F F F 22 440-459 45 45 45 71 71 71 F F F 23 460-479 42 42 42 64 64 64 F F F 24 480-499 42 42 42 64 64 64 F F F 25 500-519 40 40 40 58 58 58 F F F 26 520-539 41 41 41 59 59 59 F F F 27 540-559 45 45 45 71 71 71 F F F 28 560-579 45 45 45 71 71 71 F F F 29 580-599 36 36 36 51 51 51 F F F Max gen: 45, Max flush: 71 Begin loop 23 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.330412992Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.342083238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.342437425Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.342475711Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.342488014Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.342505842Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386} {"msg":"Crucible 66b5ae31-43e9-4928-90f6-cd37a806ef54 has session id: 069b84f0-6a24-4bdc-ae59-a23011cf4130","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.342874511Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Upstairs opts: Upstairs UUID: 66b5ae31-43e9-4928-90f6-cd37a806ef54, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.342909399Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Crucible stats registered with UUID: 66b5ae31-43e9-4928-90f6-cd37a806ef54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.342925499Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:41.34293876Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343092514Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"io task","client":"0","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343119697Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"io task","client":"2","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343134527Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"io task","client":"1","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343253472Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"task":"crutest"} {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343269103Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386} {"msg":"66b5ae31-43e9-4928-90f6-cd37a806ef54 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343286981Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343299663Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"client":"0","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343313695Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"client":"1","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343327476Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"client":"2","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343341017Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"io task","client":"2","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343355139Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"io task","client":"0","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.34336939Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"io task","client":"1","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"ds_connection connected from Ok(127.0.0.1:40299)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343393974Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"io task","client":"2","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"ds_connection connected from Ok(127.0.0.1:54628)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343408775Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"io task","client":"1","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"ds_connection connected from Ok(127.0.0.1:63206)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.343475203Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"io task","client":"0","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344202446Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"client":"1","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344231158Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"client":"0","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344251186Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"client":"2","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344574462Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344622282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"[0]R flush_numbers[0..12]: [69, 60, 65, 71, 61, 61, 61, 64, 65, 69, 69, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344678867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"[0]R generation[0..12]: [44, 42, 43, 45, 42, 42, 42, 42, 43, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344695536Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344712056Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"[1]R flush_numbers[0..12]: [69, 60, 65, 71, 61, 61, 61, 64, 65, 69, 69, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344727646Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"[1]R generation[0..12]: [44, 42, 43, 45, 42, 42, 42, 42, 43, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344742966Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344759176Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"[2]R flush_numbers[0..12]: [69, 60, 65, 71, 61, 61, 61, 64, 65, 69, 69, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344774297Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"[2]R generation[0..12]: [44, 42, 43, 45, 42, 42, 42, 42, 43, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344789757Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344805767Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344819448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairsWait for a query_work_queue command to finish before sending IO ","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344839406Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Next flush: 72","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344855726Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344868977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"":"downstairs","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344882809Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344895441Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"client":"0","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344909492Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"client":"1","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344923433Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"client":"2","":"downstairs","session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344940362Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344954374Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"66b5ae31-43e9-4928-90f6-cd37a806ef54 is now active with session: 069b84f0-6a24-4bdc-ae59-a23011cf4130","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344969194Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344981756Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386,"session_id":"069b84f0-6a24-4bdc-ae59-a23011cf4130"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.344994998Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386} {"msg":"Activated sub_volume 66b5ae31-43e9-4928-90f6-cd37a806ef54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:41.345006601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 588 len 24576 data: 4 6 6 5 4 4 02/30 Write block 308 len 40960 data: 11 9 9 8 10 9 8 9 6 4 03/30 Write block 382 len 8192 data: 3 3 04/30 Write block 68 len 40960 data: 4 4 5 5 6 6 8 8 6 5 05/30 Read block 313 len 24576 06/30 Write block 52 len 40960 data: 4 4 3 4 3 3 3 4 6 6 07/30 Write block 174 len 20480 data: 2 2 3 3 3 08/30 Read block 22 len 28672 09/30 Write block 512 len 20480 data: 6 4 4 6 4 10/30 Write block 502 len 28672 data: 3 4 4 4 4 4 6 11/30 Read block 412 len 24576 12/30 Read block 297 len 16384 13/30 Write block 99 len 4096 data: 3 14/30 Read block 342 len 24576 15/30 Read block 174 len 12288 16/30 Read block 96 len 20480 17/30 Write block 274 len 24576 data: 5 5 5 3 3 3 18/30 Read block 370 len 8192 19/30 Read block 333 len 40960 20/30 Read block 265 len 12288 21/30 Read block 230 len 36864 22/30 Write block 453 len 36864 data: 4 4 7 7 6 5 6 5 4 23/30 Read block 191 len 16384 24/30 Write block 420 len 16384 data: 5 5 4 6 25/30 Read block 404 len 24576 26/30 Read block 228 len 36864 27/30 Write block 355 len 28672 data: 3 3 3 3 2 3 4 28/30 Write block 254 len 40960 data: 4 4 5 5 5 5 5 4 4 6 29/30 Read block 111 len 32768 30/30 Read block 107 len 4096 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Write 10 Done Done Done false 1004 Acked Read 6 Done Done Done false 1005 Acked Write 10 Done Done Done false 1006 Acked Write 5 Done Done Done false 1007 Acked Read 7 Done Done Done false 1008 Acked Write 5 Done Done Sent false 1009 Acked Write 7 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Read 4 Done Done Sent false 1012 Acked Write 1 Done Done Sent false 1013 Acked Read 6 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Read 5 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Read 4 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Write 10 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 22 23 Done 30 29 8 67 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:41.35748383Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1386} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 69 69 69 F F F 1 020-039 42 42 42 60 60 60 F F F 2 040-059 43 43 43 65 65 65 T T T 3 060-079 45 45 45 71 71 71 T T T 4 080-099 42 42 42 61 61 61 T T F <--- 5 100-119 42 42 42 61 61 61 F F F 6 120-139 42 42 42 61 61 61 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 43 43 43 65 65 65 T T T 9 180-199 44 44 44 69 69 69 F F F 10 200-219 44 44 44 69 69 69 F F F 11 220-239 44 44 44 67 67 67 F F F 12 240-259 44 44 44 67 67 67 T T F <--- 13 260-279 38 38 38 54 54 54 T T F <--- 14 280-299 42 42 42 60 60 60 F F F 15 300-319 39 39 39 56 56 56 T T T 16 320-339 44 44 44 70 70 70 F F F 17 340-359 36 36 36 50 50 50 T T F <--- 18 360-379 44 44 44 70 70 70 T T F <--- 19 380-399 44 44 44 70 70 70 T T T 20 400-419 40 40 40 58 58 58 F F F 21 420-439 39 39 39 56 56 56 T T F <--- 22 440-459 45 45 45 71 71 71 T T F <--- 23 460-479 42 42 42 64 64 64 T T F <--- 24 480-499 42 42 42 64 64 64 F F F 25 500-519 40 40 40 58 58 58 T T F <--- 26 520-539 41 41 41 59 59 59 F F F 27 540-559 45 45 45 71 71 71 F F F 28 560-579 45 45 45 71 71 71 F F F 29 580-599 36 36 36 51 51 51 T T T Max gen: 45, Max flush: 71 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 69 69 69 F F F 1 020-039 42 42 42 60 60 60 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 46 46 46 72 72 72 F F F 4 080-099 46 46 46 72 72 72 F F F 5 100-119 42 42 42 61 61 61 F F F 6 120-139 42 42 42 61 61 61 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 46 46 46 72 72 72 F F F 9 180-199 44 44 44 69 69 69 F F F 10 200-219 44 44 44 69 69 69 F F F 11 220-239 44 44 44 67 67 67 F F F 12 240-259 46 46 46 72 72 72 F F F 13 260-279 46 46 46 72 72 72 F F F 14 280-299 42 42 42 60 60 60 F F F 15 300-319 46 46 46 72 72 72 F F F 16 320-339 44 44 44 70 70 70 F F F 17 340-359 46 46 46 72 72 72 F F F 18 360-379 46 46 46 72 72 72 F F F 19 380-399 46 46 46 72 72 72 F F F 20 400-419 40 40 40 58 58 58 F F F 21 420-439 46 46 46 72 72 72 F F F 22 440-459 46 46 46 72 72 72 F F F 23 460-479 46 46 46 72 72 72 F F F 24 480-499 42 42 42 64 64 64 F F F 25 500-519 46 46 46 72 72 72 F F F 26 520-539 41 41 41 59 59 59 F F F 27 540-559 45 45 45 71 71 71 F F F 28 560-579 45 45 45 71 71 71 F F F 29 580-599 46 46 46 72 72 72 F F F Max gen: 46, Max flush: 72 Begin loop 24 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.692939479Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.704565153Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.704880585Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.704916462Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.704932132Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.704950121Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394} {"msg":"Crucible bc293c9a-09e7-4187-985d-1d597aa60fd8 has session id: fec058f8-bb52-4e7d-b969-cb5b246bbe5c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705274407Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Upstairs opts: Upstairs UUID: bc293c9a-09e7-4187-985d-1d597aa60fd8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705322117Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Crucible stats registered with UUID: bc293c9a-09e7-4187-985d-1d597aa60fd8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705338137Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:45.705351439Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705511209Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"io task","client":"0","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705533974Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"io task","client":"1","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705548485Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"io task","client":"2","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705675545Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"task":"crutest"} {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705690536Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394} {"msg":"bc293c9a-09e7-4187-985d-1d597aa60fd8 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705711523Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705724295Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"client":"0","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705738216Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"client":"1","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705752037Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"client":"2","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705767967Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"io task","client":"2","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705782068Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"io task","client":"0","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.705796189Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"io task","client":"1","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"ds_connection connected from Ok(127.0.0.1:33119)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.70590689Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"io task","client":"0","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"ds_connection connected from Ok(127.0.0.1:65228)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.70598604Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"io task","client":"2","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"ds_connection connected from Ok(127.0.0.1:57467)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.706015102Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"io task","client":"1","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.706594048Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"client":"0","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.706635932Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"client":"1","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.706729213Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"client":"2","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.706969522Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707011286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"[0]R flush_numbers[0..12]: [69, 60, 72, 72, 72, 61, 61, 64, 72, 69, 69, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.70706818Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"[0]R generation[0..12]: [44, 42, 46, 46, 46, 42, 42, 42, 46, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707090976Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707107905Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"[1]R flush_numbers[0..12]: [69, 60, 72, 72, 72, 61, 61, 64, 72, 69, 69, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707123445Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"[1]R generation[0..12]: [44, 42, 46, 46, 46, 42, 42, 42, 46, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707141894Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707158483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"[2]R flush_numbers[0..12]: [69, 60, 72, 72, 72, 61, 61, 64, 72, 69, 69, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707173844Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"[2]R generation[0..12]: [44, 42, 46, 46, 46, 42, 42, 42, 46, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707189124Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707205194Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Max found gen is 47","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:02:45.707218416Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Generation requested: 48 >= found:47","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707237264Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Next flush: 73","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707250496Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707263737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"":"downstairs","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707278158Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.70729058Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"client":"0","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707304901Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"client":"1","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707318823Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"client":"2","":"downstairs","session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707335582Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707349144Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"bc293c9a-09e7-4187-985d-1d597aa60fd8 is now active with session: fec058f8-bb52-4e7d-b969-cb5b246bbe5c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707366053Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707378595Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394,"session_id":"fec058f8-bb52-4e7d-b969-cb5b246bbe5c"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707392087Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394} {"msg":"Activated sub_volume bc293c9a-09e7-4187-985d-1d597aa60fd8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:45.707403479Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 123 len 16384 data: 3 5 4 4 02/30 Flush 03/30 Flush 04/30 Write block 458 len 24576 data: 6 7 6 5 4 3 05/30 Read block 395 len 12288 06/30 Read block 245 len 28672 07/30 Write block 109 len 12288 data: 5 5 4 08/30 Write block 28 len 40960 data: 3 3 4 4 3 4 4 5 5 4 09/30 Write block 103 len 20480 data: 5 5 5 5 4 10/30 Write block 356 len 20480 data: 4 4 4 3 4 11/30 Write block 366 len 24576 data: 6 7 6 6 4 3 12/30 Write block 274 len 32768 data: 6 6 6 4 4 4 3 2 13/30 Read block 333 len 40960 14/30 Write block 248 len 28672 data: 9 7 7 7 4 4 5 15/30 Read block 243 len 16384 16/30 Write block 578 len 20480 data: 7 6 8 8 8 17/30 Read block 236 len 24576 18/30 Write block 261 len 20480 data: 5 5 7 6 6 19/30 Read block 6 len 12288 20/30 Write block 289 len 16384 data: 6 5 7 7 21/30 Read block 391 len 20480 22/30 Write block 263 len 40960 data: 8 7 7 6 5 5 5 7 6 5 23/30 Write block 539 len 16384 data: 2 4 4 3 24/30 Read block 448 len 12288 25/30 Write block 337 len 8192 data: 2 2 26/30 Write block 319 len 32768 data: 3 4 3 3 3 4 4 5 27/30 Write block 109 len 28672 data: 6 6 5 5 5 4 3 28/30 Write block 324 len 32768 data: 5 5 6 6 6 5 6 6 29/30 Write block 402 len 28672 data: 4 3 3 3 3 4 4 30/30 Write block 74 len 36864 data: 9 9 7 6 6 7 5 5 4 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 6 Done Done Sent false 1004 Acked Read 3 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Write 3 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Write 5 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Write 4 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Write 7 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 27 29 Done 26 26 0 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1002)) Some(JobId(1002)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:45.719126343Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1394} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 69 69 69 F F F 1 020-039 42 42 42 60 60 60 T T F <--- 2 040-059 46 46 46 72 72 72 F F F 3 060-079 46 46 46 72 72 72 T T F <--- 4 080-099 46 46 46 72 72 72 T T F <--- 5 100-119 42 42 42 61 61 61 T T F <--- 6 120-139 48 48 48 73 73 73 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 46 46 46 72 72 72 F F F 9 180-199 44 44 44 69 69 69 F F F 10 200-219 44 44 44 69 69 69 F F F 11 220-239 44 44 44 67 67 67 F F F 12 240-259 46 46 46 72 72 72 T T F <--- 13 260-279 46 46 46 72 72 72 T T F <--- 14 280-299 42 42 42 60 60 60 T T F <--- 15 300-319 46 46 46 72 72 72 T T F <--- 16 320-339 44 44 44 70 70 70 T T F <--- 17 340-359 46 46 46 72 72 72 T T F <--- 18 360-379 46 46 46 72 72 72 T T F <--- 19 380-399 46 46 46 72 72 72 F F F 20 400-419 40 40 40 58 58 58 T T F <--- 21 420-439 46 46 46 72 72 72 F F F 22 440-459 46 46 46 72 72 72 T T F <--- 23 460-479 46 46 46 72 72 72 T T F <--- 24 480-499 42 42 42 64 64 64 F F F 25 500-519 46 46 46 72 72 72 F F F 26 520-539 41 41 41 59 59 59 T T F <--- 27 540-559 45 45 45 71 71 71 T T F <--- 28 560-579 45 45 45 71 71 71 T T F <--- 29 580-599 46 46 46 72 72 72 T T F <--- Max gen: 48, Max flush: 73 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 24 Verifying data now No change after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 69 69 69 F F F 1 020-039 49 49 49 74 74 74 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 49 49 49 74 74 74 F F F 4 080-099 49 49 49 74 74 74 F F F 5 100-119 49 49 49 74 74 74 F F F 6 120-139 48 48 48 73 73 73 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 46 46 46 72 72 72 F F F 9 180-199 44 44 44 69 69 69 F F F 10 200-219 44 44 44 69 69 69 F F F 11 220-239 44 44 44 67 67 67 F F F 12 240-259 49 49 49 74 74 74 F F F 13 260-279 49 49 49 74 74 74 F F F 14 280-299 49 49 49 74 74 74 F F F 15 300-319 49 49 49 74 74 74 F F F 16 320-339 49 49 49 74 74 74 F F F 17 340-359 49 49 49 74 74 74 F F F 18 360-379 49 49 49 74 74 74 F F F 19 380-399 46 46 46 72 72 72 F F F 20 400-419 49 49 49 74 74 74 F F F 21 420-439 46 46 46 72 72 72 F F F 22 440-459 49 49 49 74 74 74 F F F 23 460-479 49 49 49 74 74 74 F F F 24 480-499 42 42 42 64 64 64 F F F 25 500-519 46 46 46 72 72 72 F F F 26 520-539 49 49 49 74 74 74 F F F 27 540-559 49 49 49 74 74 74 F F F 28 560-579 49 49 49 74 74 74 F F F 29 580-599 49 49 49 74 74 74 F F F Max gen: 49, Max flush: 74 Begin loop 25 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.059534728Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:50.070147399Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.070203634Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.080559016Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.080847035Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.080884752Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.080896684Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.080914653Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402} {"msg":"Crucible 87997c50-7a4f-4854-8a49-d543d29f4539 has session id: e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081281222Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Upstairs opts: Upstairs UUID: 87997c50-7a4f-4854-8a49-d543d29f4539, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081307506Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Crucible stats registered with UUID: 87997c50-7a4f-4854-8a49-d543d29f4539","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081322387Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:50.081338127Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081504372Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"io task","client":"1","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081528597Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"io task","client":"0","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081543638Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"io task","client":"2","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081670048Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"task":"crutest"} {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081686358Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402} {"msg":"87997c50-7a4f-4854-8a49-d543d29f4539 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081702867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081715599Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"client":"0","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081729401Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"client":"1","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081743012Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"client":"2","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081756674Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"io task","client":"2","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081770985Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"io task","client":"0","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081785535Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"io task","client":"1","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"ds_connection connected from Ok(127.0.0.1:57869)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081835474Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"io task","client":"0","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"ds_connection connected from Ok(127.0.0.1:49768)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081863297Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"io task","client":"1","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"ds_connection connected from Ok(127.0.0.1:46369)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.081947114Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"io task","client":"2","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.082617223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"client":"2","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.082654489Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"client":"1","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.082678155Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"client":"0","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.082979195Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.082992897Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"[0]R flush_numbers[0..12]: [69, 74, 72, 74, 74, 74, 73, 64, 72, 69, 69, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083117219Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"[0]R generation[0..12]: [44, 49, 46, 49, 49, 49, 48, 42, 46, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083133978Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083150568Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"[1]R flush_numbers[0..12]: [69, 74, 72, 74, 74, 74, 73, 64, 72, 69, 69, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083165838Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"[1]R generation[0..12]: [44, 49, 46, 49, 49, 49, 48, 42, 46, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083181079Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083197508Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"[2]R flush_numbers[0..12]: [69, 74, 72, 74, 74, 74, 73, 64, 72, 69, 69, 67]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083212609Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"[2]R generation[0..12]: [44, 49, 46, 49, 49, 49, 48, 42, 46, 44, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083227809Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083243939Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083260189Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:02:50.08327442Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Next flush: 75","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083292139Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.08330562Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"":"downstairs","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083319741Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083332543Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"client":"0","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083346674Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"client":"1","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083360666Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"client":"2","":"downstairs","session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083375037Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083391246Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"87997c50-7a4f-4854-8a49-d543d29f4539 is now active with session: e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083406117Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083418849Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402,"session_id":"e4ffbc62-a6c0-4bb0-b3ae-a2ad0babbac1"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083432151Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402} {"msg":"Activated sub_volume 87997c50-7a4f-4854-8a49-d543d29f4539","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:50.083443574Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 239 len 8192 data: 5 5 02/30 Write block 90 len 8192 data: 7 7 03/30 Read block 23 len 24576 04/30 Read block 0 len 8192 05/30 Read block 15 len 4096 06/30 Write block 241 len 20480 data: 4 5 6 7 8 07/30 Write block 69 len 12288 data: 5 6 6 08/30 Read block 506 len 8192 09/30 Read block 127 len 8192 10/30 Write block 3 len 20480 data: 5 4 5 4 4 11/30 Read block 93 len 28672 12/30 Read block 16 len 20480 13/30 Write block 489 len 8192 data: 5 5 14/30 Read block 349 len 4096 15/30 Read block 538 len 12288 16/30 Write block 9 len 28672 data: 3 3 3 3 3 3 4 17/30 Read block 280 len 24576 18/30 Read block 37 len 12288 19/30 Write block 177 len 28672 data: 4 4 3 2 2 4 6 20/30 Read block 447 len 24576 21/30 Read block 567 len 20480 22/30 Read block 581 len 32768 23/30 Read block 4 len 32768 24/30 Write block 434 len 4096 data: 6 25/30 Read block 327 len 4096 26/30 Read block 343 len 12288 27/30 Read block 116 len 8192 28/30 Write block 367 len 8192 data: 8 7 29/30 Read block 555 len 36864 30/30 Read block 217 len 40960 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 2 Sent Done Done false 1002 Acked Read 6 Sent Done Done false 1003 Acked Read 2 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Read 2 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Read 5 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Read 10 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 1 30 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:50.09422237Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1402} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 69 69 69 F T T <--- 1 020-039 49 49 49 74 74 74 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 49 49 49 74 74 74 F T T <--- 4 080-099 49 49 49 74 74 74 F T T <--- 5 100-119 49 49 49 74 74 74 F F F 6 120-139 48 48 48 73 73 73 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 46 46 46 72 72 72 F T T <--- 9 180-199 44 44 44 69 69 69 F T T <--- 10 200-219 44 44 44 69 69 69 F F F 11 220-239 44 44 44 67 67 67 T T T 12 240-259 49 49 49 74 74 74 T T T 13 260-279 49 49 49 74 74 74 F F F 14 280-299 49 49 49 74 74 74 F F F 15 300-319 49 49 49 74 74 74 F F F 16 320-339 49 49 49 74 74 74 F F F 17 340-359 49 49 49 74 74 74 F F F 18 360-379 49 49 49 74 74 74 F T T <--- 19 380-399 46 46 46 72 72 72 F F F 20 400-419 49 49 49 74 74 74 F F F 21 420-439 46 46 46 72 72 72 F T T <--- 22 440-459 49 49 49 74 74 74 F F F 23 460-479 49 49 49 74 74 74 F F F 24 480-499 42 42 42 64 64 64 F T T <--- 25 500-519 46 46 46 72 72 72 F F F 26 520-539 49 49 49 74 74 74 F F F 27 540-559 49 49 49 74 74 74 F F F 28 560-579 49 49 49 74 74 74 F F F 29 580-599 49 49 49 74 74 74 F F F Max gen: 49, Max flush: 74 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 25 Verifying data now Files /tmp/test_live_repair/test_repair_verify.data and /tmp/test_live_repair/test_repair_verify.data.last differ 244d243 < 3, 248a248 > 8, diff found after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 75 75 75 F F F 1 020-039 49 49 49 74 74 74 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 49 49 49 74 74 74 F F F 6 120-139 48 48 48 73 73 73 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 50 50 50 75 75 75 F F F 9 180-199 50 50 50 75 75 75 F F F 10 200-219 44 44 44 69 69 69 F F F 11 220-239 50 50 50 75 75 75 F F F 12 240-259 50 50 50 75 75 75 F F F 13 260-279 49 49 49 74 74 74 F F F 14 280-299 49 49 49 74 74 74 F F F 15 300-319 49 49 49 74 74 74 F F F 16 320-339 49 49 49 74 74 74 F F F 17 340-359 49 49 49 74 74 74 F F F 18 360-379 50 50 50 75 75 75 F F F 19 380-399 46 46 46 72 72 72 F F F 20 400-419 49 49 49 74 74 74 F F F 21 420-439 50 50 50 75 75 75 F F F 22 440-459 49 49 49 74 74 74 F F F 23 460-479 49 49 49 74 74 74 F F F 24 480-499 50 50 50 75 75 75 F F F 25 500-519 46 46 46 72 72 72 F F F 26 520-539 49 49 49 74 74 74 F F F 27 540-559 49 49 49 74 74 74 F F F 28 560-579 49 49 49 74 74 74 F F F 29 580-599 49 49 49 74 74 74 F F F Max gen: 50, Max flush: 75 Begin loop 26 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.429179442Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.440871585Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441185318Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441224933Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441240243Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441258142Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413} {"msg":"Crucible 1cd03318-2d89-46c5-b7c1-503a2b144da4 has session id: 65c4ca12-86ad-4caa-b601-70109749be7e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441561551Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Upstairs opts: Upstairs UUID: 1cd03318-2d89-46c5-b7c1-503a2b144da4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441597968Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Crucible stats registered with UUID: 1cd03318-2d89-46c5-b7c1-503a2b144da4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441612509Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:54.441625921Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441790487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"io task","client":"1","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441814272Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"io task","client":"0","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441829123Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"io task","client":"2","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441960001Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"task":"crutest"} {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.44197664Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413} {"msg":"1cd03318-2d89-46c5-b7c1-503a2b144da4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.441997847Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.442015296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"client":"0","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.442029427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"client":"1","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.442045607Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"client":"2","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.442059598Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"io task","client":"0","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.44207361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"io task","client":"1","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.442111736Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"io task","client":"2","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"ds_connection connected from Ok(127.0.0.1:56702)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.44213664Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"io task","client":"0","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"ds_connection connected from Ok(127.0.0.1:36916)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.442268697Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"io task","client":"1","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"ds_connection connected from Ok(127.0.0.1:46878)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.442287615Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"io task","client":"2","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.442901599Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"client":"1","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.442924225Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"client":"2","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.442945472Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"client":"0","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443284029Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443298939Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"[0]R flush_numbers[0..12]: [75, 74, 72, 75, 75, 74, 73, 64, 75, 75, 69, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443331929Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"[0]R generation[0..12]: [50, 49, 46, 50, 50, 49, 48, 42, 50, 50, 44, 50]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443348698Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443367786Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"[1]R flush_numbers[0..12]: [75, 74, 72, 75, 75, 74, 73, 64, 75, 75, 69, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443383276Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"[1]R generation[0..12]: [50, 49, 46, 50, 50, 49, 48, 42, 50, 50, 44, 50]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443398687Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443414927Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"[2]R flush_numbers[0..12]: [75, 74, 72, 75, 75, 74, 73, 64, 75, 75, 69, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443430047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"[2]R generation[0..12]: [50, 49, 46, 50, 50, 49, 48, 42, 50, 50, 44, 50]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443445088Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443461218Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Max found gen is 51","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-03-14T05:02:54.443480415Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Generation requested: 52 >= found:51","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443494447Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Next flush: 76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443507708Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.44352091Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"":"downstairs","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443534721Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443547343Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"client":"0","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443561784Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"client":"1","":"downstairs","session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443575786Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443595013Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443608205Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"1cd03318-2d89-46c5-b7c1-503a2b144da4 is now active with session: 65c4ca12-86ad-4caa-b601-70109749be7e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443623286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443635898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413,"session_id":"65c4ca12-86ad-4caa-b601-70109749be7e"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443649259Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413} {"msg":"Activated sub_volume 1cd03318-2d89-46c5-b7c1-503a2b144da4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:54.443660692Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 313 len 20480 data: 10 9 10 7 5 02/30 Write block 360 len 20480 data: 5 5 5 3 3 03/30 Read block 118 len 8192 04/30 Read block 220 len 12288 05/30 Read block 551 len 16384 06/30 Read block 515 len 16384 07/30 Write block 402 len 4096 data: 5 08/30 Write block 257 len 20480 data: 6 6 6 6 6 09/30 Read block 66 len 32768 10/30 Write block 477 len 28672 data: 5 7 8 9 7 6 5 11/30 Read block 41 len 32768 12/30 Read block 475 len 32768 13/30 Write block 498 len 40960 data: 2 2 2 2 4 5 5 5 5 5 14/30 Write block 323 len 12288 data: 4 6 6 15/30 Flush 16/30 Flush 17/30 Write block 434 len 12288 data: 7 7 7 18/30 Read block 462 len 8192 19/30 Write block 325 len 16384 data: 7 7 7 7 20/30 Write block 335 len 32768 data: 2 3 3 3 2 2 2 4 21/30 Flush 22/30 Flush 23/30 Write block 401 len 32768 data: 5 6 4 4 4 4 5 5 24/30 Read block 13 len 40960 25/30 Read block 540 len 4096 26/30 Read block 393 len 12288 27/30 Read block 331 len 8192 28/30 Write block 387 len 28672 data: 4 5 5 4 3 3 4 29/30 Read block 197 len 12288 30/30 Read block 552 len 20480 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Read 4 Done Done Done false 1005 Acked Read 4 Done Done Done false 1006 Acked Write 1 Done Done Done false 1007 Acked Write 5 Done Done Done false 1008 Acked Read 8 Done Done Done false 1009 Acked Write 7 Done Done Done false 1010 Acked Read 8 Done Done Done false 1011 Acked Read 8 Done Done Done false 1012 Acked Write 10 Done Done Done false 1013 Acked Write 3 Done Done Done false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Write 7 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Read 5 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 16 17 Done 29 30 14 73 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:54.455864053Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1413} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 75 75 75 F F F 1 020-039 49 49 49 74 74 74 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 49 49 49 74 74 74 F F F 6 120-139 48 48 48 73 73 73 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 50 50 50 75 75 75 F F F 9 180-199 50 50 50 75 75 75 F F F 10 200-219 44 44 44 69 69 69 F F F 11 220-239 50 50 50 75 75 75 F F F 12 240-259 52 52 50 76 76 75 F F T <--- 13 260-279 52 52 49 76 76 74 F F T <--- 14 280-299 49 49 49 74 74 74 F F F 15 300-319 52 52 49 76 76 74 F F T <--- 16 320-339 52 52 49 78 78 74 F F T <--- 17 340-359 52 52 49 78 78 74 F F F <--- 18 360-379 52 52 50 76 76 75 F F T <--- 19 380-399 46 46 46 72 72 72 T T F <--- 20 400-419 52 52 49 76 76 74 T T T <--- 21 420-439 52 52 50 78 78 75 F F F <--- 22 440-459 49 49 49 74 74 74 F F F 23 460-479 52 52 49 76 76 74 F F T <--- 24 480-499 52 52 50 76 76 75 F F T <--- 25 500-519 52 52 46 76 76 72 F F T <--- 26 520-539 49 49 49 74 74 74 F F F 27 540-559 49 49 49 74 74 74 F F F 28 560-579 49 49 49 74 74 74 F F F 29 580-599 49 49 49 74 74 74 F F F Max gen: 52, Max flush: 78 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 75 75 75 F F F 1 020-039 49 49 49 74 74 74 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 49 49 49 74 74 74 F F F 6 120-139 48 48 48 73 73 73 F F F 7 140-159 42 42 42 64 64 64 F F F 8 160-179 50 50 50 75 75 75 F F F 9 180-199 50 50 50 75 75 75 F F F 10 200-219 44 44 44 69 69 69 F F F 11 220-239 50 50 50 75 75 75 F F F 12 240-259 52 52 52 76 76 76 F F F 13 260-279 52 52 52 76 76 76 F F F 14 280-299 49 49 49 74 74 74 F F F 15 300-319 52 52 52 76 76 76 F F F 16 320-339 52 52 52 78 78 78 F F F 17 340-359 52 52 52 78 78 78 F F F 18 360-379 52 52 52 76 76 76 F F F 19 380-399 53 53 53 79 79 79 F F F 20 400-419 53 53 53 79 79 79 F F F 21 420-439 52 52 52 78 78 78 F F F 22 440-459 49 49 49 74 74 74 F F F 23 460-479 52 52 52 76 76 76 F F F 24 480-499 52 52 52 76 76 76 F F F 25 500-519 52 52 52 76 76 76 F F F 26 520-539 49 49 49 74 74 74 F F F 27 540-559 49 49 49 74 74 74 F F F 28 560-579 49 49 49 74 74 74 F F F 29 580-599 49 49 49 74 74 74 F F F Max gen: 53, Max flush: 79 Begin loop 27 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.771833147Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.783433047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.78374648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.783784067Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.783796399Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.783814008Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421} {"msg":"Crucible 134340af-3461-4fb8-81d9-f88f91b37ab4 has session id: b9e48a2f-225f-4b7b-8fc6-a42ad4d86997","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784198646Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Upstairs opts: Upstairs UUID: 134340af-3461-4fb8-81d9-f88f91b37ab4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784233864Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Crucible stats registered with UUID: 134340af-3461-4fb8-81d9-f88f91b37ab4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784250324Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:58.784266923Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784413831Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"io task","client":"2","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784438665Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"io task","client":"0","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784453656Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"io task","client":"1","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784552334Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"task":"crutest"} {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784568734Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421} {"msg":"134340af-3461-4fb8-81d9-f88f91b37ab4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.78459103Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784604121Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"client":"0","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784618103Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"client":"1","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784631834Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"client":"2","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784645925Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"io task","client":"2","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784660546Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"io task","client":"1","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784674527Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"io task","client":"0","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"ds_connection connected from Ok(127.0.0.1:37725)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784699392Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"io task","client":"1","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"ds_connection connected from Ok(127.0.0.1:40693)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784714432Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"io task","client":"0","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"ds_connection connected from Ok(127.0.0.1:41056)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.784729513Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"io task","client":"2","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785382182Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"client":"0","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785412174Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"client":"1","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785499249Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"client":"2","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785726906Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785764952Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"[0]R flush_numbers[0..12]: [75, 74, 72, 75, 75, 74, 73, 64, 75, 75, 69, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785874983Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"[0]R generation[0..12]: [50, 49, 46, 50, 50, 49, 48, 42, 50, 50, 44, 50]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785891802Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785908422Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"[1]R flush_numbers[0..12]: [75, 74, 72, 75, 75, 74, 73, 64, 75, 75, 69, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785923782Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"[1]R generation[0..12]: [50, 49, 46, 50, 50, 49, 48, 42, 50, 50, 44, 50]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785939082Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785955142Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"[2]R flush_numbers[0..12]: [75, 74, 72, 75, 75, 74, 73, 64, 75, 75, 69, 75]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785970683Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"[2]R generation[0..12]: [50, 49, 46, 50, 50, 49, 48, 42, 50, 50, 44, 50]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.785986003Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786002213Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786015914Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786032094Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Next flush: 80","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786045685Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} Wait for a query_work_queue command to finish before sending IO {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786064134Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"":"downstairs","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786078705Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786091647Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"client":"0","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786105848Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"client":"1","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786119779Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"client":"2","":"downstairs","session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.78613388Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786147472Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"134340af-3461-4fb8-81d9-f88f91b37ab4 is now active with session: b9e48a2f-225f-4b7b-8fc6-a42ad4d86997","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.78616505Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786177802Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421,"session_id":"b9e48a2f-225f-4b7b-8fc6-a42ad4d86997"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786191234Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421} {"msg":"Activated sub_volume 134340af-3461-4fb8-81d9-f88f91b37ab4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:02:58.786202517Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 441 len 32768 data: 6 7 8 8 8 8 6 4 02/30 Read block 499 len 12288 03/30 Read block 464 len 40960 04/30 Read block 434 len 28672 05/30 Write block 442 len 16384 data: 8 9 9 9 06/30 Write block 256 len 28672 data: 6 7 7 7 7 7 6 07/30 Write block 155 len 24576 data: 4 4 4 5 5 4 08/30 Read block 176 len 8192 09/30 Flush 10/30 Write block 297 len 8192 data: 4 4 11/30 Read block 240 len 8192 12/30 Read block 177 len 24576 13/30 Read block 578 len 16384 14/30 Read block 290 len 36864 15/30 Write block 190 len 24576 data: 5 5 4 4 4 4 16/30 Flush 17/30 Write block 517 len 12288 data: 3 2 2 18/30 Read block 486 len 8192 19/30 Write block 550 len 36864 data: 5 5 5 5 6 6 7 8 9 20/30 Write block 519 len 8192 data: 3 2 21/30 Write block 235 len 40960 data: 7 6 5 6 6 6 4 5 6 7 22/30 Read block 321 len 28672 23/30 Write block 119 len 36864 data: 3 3 2 3 4 6 5 5 4 24/30 Write block 561 len 8192 data: 5 4 25/30 Read block 12 len 32768 26/30 Read block 28 len 36864 27/30 Write block 526 len 28672 data: 3 5 5 5 6 6 8 28/30 Read block 537 len 32768 29/30 Write block 514 len 8192 data: 5 7 30/30 Write block 509 len 20480 data: 6 5 4 7 5 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Read 7 Done Done Sent false 1004 Acked Write 4 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Read 2 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 2 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Read 9 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Read 8 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Write 2 Done Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 2 27 30 Done 29 28 3 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1015)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:02:58.798469927Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1421} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 75 75 75 F F F 1 020-039 49 49 49 74 74 74 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 49 49 49 74 74 74 T T F <--- 6 120-139 48 48 48 73 73 73 T T F <--- 7 140-159 54 54 42 80 80 64 F F F <--- 8 160-179 54 54 50 80 80 75 F F F <--- 9 180-199 54 54 50 81 81 75 F F F <--- 10 200-219 44 44 44 69 69 69 F F F 11 220-239 50 50 50 75 75 75 T T F <--- 12 240-259 54 54 52 80 80 76 T T F <--- 13 260-279 54 54 52 80 80 76 F F F <--- 14 280-299 54 54 49 81 81 74 F F F <--- 15 300-319 52 52 52 76 76 76 F F F 16 320-339 52 52 52 78 78 78 F F F 17 340-359 52 52 52 78 78 78 F F F 18 360-379 52 52 52 76 76 76 F F F 19 380-399 53 53 53 79 79 79 F F F 20 400-419 53 53 53 79 79 79 F F F 21 420-439 52 52 52 78 78 78 F F F 22 440-459 54 54 49 80 80 74 F F T <--- 23 460-479 52 52 52 76 76 76 F F F 24 480-499 52 52 52 76 76 76 F F F 25 500-519 52 52 52 76 76 76 T T F <--- 26 520-539 49 49 49 74 74 74 T T F <--- 27 540-559 49 49 49 74 74 74 T T F <--- 28 560-579 49 49 49 74 74 74 T T F <--- 29 580-599 49 49 49 74 74 74 F F F Max gen: 54, Max flush: 81 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 75 75 75 F F F 1 020-039 49 49 49 74 74 74 F F F 2 040-059 46 46 46 72 72 72 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 55 55 55 82 82 82 F F F 6 120-139 55 55 55 82 82 82 F F F 7 140-159 54 54 54 80 80 80 F F F 8 160-179 54 54 54 80 80 80 F F F 9 180-199 54 54 54 81 81 81 F F F 10 200-219 44 44 44 69 69 69 F F F 11 220-239 55 55 55 82 82 82 F F F 12 240-259 55 55 55 82 82 82 F F F 13 260-279 54 54 54 80 80 80 F F F 14 280-299 54 54 54 81 81 81 F F F 15 300-319 52 52 52 76 76 76 F F F 16 320-339 52 52 52 78 78 78 F F F 17 340-359 52 52 52 78 78 78 F F F 18 360-379 52 52 52 76 76 76 F F F 19 380-399 53 53 53 79 79 79 F F F 20 400-419 53 53 53 79 79 79 F F F 21 420-439 52 52 52 78 78 78 F F F 22 440-459 54 54 54 80 80 80 F F F 23 460-479 52 52 52 76 76 76 F F F 24 480-499 52 52 52 76 76 76 F F F 25 500-519 55 55 55 82 82 82 F F F 26 520-539 55 55 55 82 82 82 F F F 27 540-559 55 55 55 82 82 82 F F F 28 560-579 55 55 55 82 82 82 F F F 29 580-599 49 49 49 74 74 74 F F F Max gen: 55, Max flush: 82 Begin loop 28 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.123303641Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:03.133877176Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.133930003Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.144124537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.144452011Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.144492246Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.144504148Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.144521737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429} {"msg":"Crucible 9fcfb4d8-f7f3-45af-bab0-4330fa51effb has session id: 3e2d89e5-cdf0-4523-9788-89e90686dd76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.14486739Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Upstairs opts: Upstairs UUID: 9fcfb4d8-f7f3-45af-bab0-4330fa51effb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.144901479Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Crucible stats registered with UUID: 9fcfb4d8-f7f3-45af-bab0-4330fa51effb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.144916939Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:03.1449305Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145080316Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"io task","client":"1","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145104091Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"io task","client":"0","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145118572Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"io task","client":"2","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145227144Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"task":"crutest"} {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145243444Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429} {"msg":"9fcfb4d8-f7f3-45af-bab0-4330fa51effb active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145273405Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145289535Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"client":"0","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145303766Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"client":"1","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145317297Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"client":"2","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145331049Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"io task","client":"2","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.14534537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"io task","client":"1","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145378509Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"io task","client":"0","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"ds_connection connected from Ok(127.0.0.1:52529)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145450643Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"io task","client":"2","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"ds_connection connected from Ok(127.0.0.1:54004)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.145484052Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"io task","client":"1","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"ds_connection connected from Ok(127.0.0.1:45146)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.14556694Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"io task","client":"0","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.14623562Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"client":"1","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146263013Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"client":"2","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146279832Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"client":"0","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146609085Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146651959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"[0]R flush_numbers[0..12]: [75, 74, 72, 75, 75, 82, 82, 80, 80, 81, 69, 82]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146712381Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"[0]R generation[0..12]: [50, 49, 46, 50, 50, 55, 55, 54, 54, 54, 44, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146732218Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146748828Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"[1]R flush_numbers[0..12]: [75, 74, 72, 75, 75, 82, 82, 80, 80, 81, 69, 82]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146764338Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"[1]R generation[0..12]: [50, 49, 46, 50, 50, 55, 55, 54, 54, 54, 44, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146779838Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146796238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"[2]R flush_numbers[0..12]: [75, 74, 72, 75, 75, 82, 82, 80, 80, 81, 69, 82]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146811198Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"[2]R generation[0..12]: [50, 49, 46, 50, 50, 55, 55, 54, 54, 54, 44, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146826379Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146842599Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:03:03.146855631Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146874928Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Next flush: 83","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.14688835Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146901612Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"":"downstairs","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146915963Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146928505Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"client":"0","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146942566Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"client":"1","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146958376Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"client":"2","":"downstairs","session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:03:03.146972427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.146988617Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"9fcfb4d8-f7f3-45af-bab0-4330fa51effb is now active with session: 3e2d89e5-cdf0-4523-9788-89e90686dd76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.147003518Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.14701638Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429,"session_id":"3e2d89e5-cdf0-4523-9788-89e90686dd76"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.147029811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429} {"msg":"Activated sub_volume 9fcfb4d8-f7f3-45af-bab0-4330fa51effb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:03.147041364Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 334 len 24576 data: 3 3 4 4 4 3 02/30 Write block 49 len 24576 data: 3 3 3 5 5 4 03/30 Write block 407 len 4096 data: 6 04/30 Read block 143 len 4096 05/30 Write block 23 len 12288 data: 3 3 2 06/30 Read block 438 len 32768 07/30 Write block 553 len 36864 data: 6 7 7 8 9 10 9 9 6 08/30 Read block 32 len 20480 09/30 Write block 484 len 20480 data: 4 3 2 5 5 10/30 Write block 437 len 24576 data: 7 7 7 7 7 9 11/30 Read block 328 len 40960 12/30 Flush 13/30 Write block 42 len 16384 data: 4 3 3 3 14/30 Flush 15/30 Write block 35 len 28672 data: 6 6 5 3 4 4 4 16/30 Write block 199 len 28672 data: 4 4 5 5 6 5 7 17/30 Flush 18/30 Write block 594 len 20480 data: 3 3 3 3 2 19/30 Read block 370 len 40960 20/30 Read block 208 len 36864 21/30 Write block 28 len 36864 data: 4 4 5 5 4 5 5 7 7 22/30 Read block 489 len 40960 23/30 Read block 97 len 12288 24/30 Write block 368 len 16384 data: 8 7 5 4 25/30 Read block 289 len 36864 26/30 Read block 457 len 28672 27/30 Read block 18 len 12288 28/30 Read block 154 len 32768 29/30 Read block 284 len 20480 30/30 Write block 13 len 16384 data: 4 4 5 4 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Write 3 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Write 9 Sent Done Done false 1007 Acked Read 5 Sent Done Done false 1008 Acked Write 5 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Read 10 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 1 1 29 Done 3 29 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:03.159802534Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1429} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 50 50 50 75 75 75 F T T <--- 1 020-039 49 56 56 74 85 85 F T T <--- 2 040-059 46 56 56 72 85 85 T F F <--- 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 55 55 55 82 82 82 F F F 6 120-139 55 55 55 82 82 82 F F F 7 140-159 54 54 54 80 80 80 F F F 8 160-179 54 54 54 80 80 80 F F F 9 180-199 54 56 56 81 85 85 F F F <--- 10 200-219 44 56 56 69 85 85 F F F <--- 11 220-239 55 55 55 82 82 82 F F F 12 240-259 55 55 55 82 82 82 F F F 13 260-279 54 54 54 80 80 80 F F F 14 280-299 54 54 54 81 81 81 F F F 15 300-319 52 52 52 76 76 76 F F F 16 320-339 52 56 56 78 83 83 T F F <--- 17 340-359 52 52 52 78 78 78 F F F 18 360-379 52 52 52 76 76 76 F T T <--- 19 380-399 53 53 53 79 79 79 F F F 20 400-419 53 56 56 79 83 83 T F F <--- 21 420-439 52 56 56 78 83 83 F F F <--- 22 440-459 54 56 56 80 83 83 F F F <--- 23 460-479 52 52 52 76 76 76 F F F 24 480-499 52 56 56 76 83 83 F F F <--- 25 500-519 55 55 55 82 82 82 F F F 26 520-539 55 55 55 82 82 82 F F F 27 540-559 55 56 56 82 83 83 F F F <--- 28 560-579 55 56 56 82 83 83 F F F <--- 29 580-599 49 49 49 74 74 74 F T T <--- Max gen: 56, Max flush: 85 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 86 86 86 F F F 1 020-039 57 57 57 86 86 86 F F F 2 040-059 56 56 56 85 85 85 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 55 55 55 82 82 82 F F F 6 120-139 55 55 55 82 82 82 F F F 7 140-159 54 54 54 80 80 80 F F F 8 160-179 54 54 54 80 80 80 F F F 9 180-199 56 56 56 85 85 85 F F F 10 200-219 56 56 56 85 85 85 F F F 11 220-239 55 55 55 82 82 82 F F F 12 240-259 55 55 55 82 82 82 F F F 13 260-279 54 54 54 80 80 80 F F F 14 280-299 54 54 54 81 81 81 F F F 15 300-319 52 52 52 76 76 76 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 52 52 52 78 78 78 F F F 18 360-379 57 57 57 86 86 86 F F F 19 380-399 53 53 53 79 79 79 F F F 20 400-419 56 56 56 83 83 83 F F F 21 420-439 56 56 56 83 83 83 F F F 22 440-459 56 56 56 83 83 83 F F F 23 460-479 52 52 52 76 76 76 F F F 24 480-499 56 56 56 83 83 83 F F F 25 500-519 55 55 55 82 82 82 F F F 26 520-539 55 55 55 82 82 82 F F F 27 540-559 56 56 56 83 83 83 F F F 28 560-579 56 56 56 83 83 83 F F F 29 580-599 57 57 57 86 86 86 F F F Max gen: 57, Max flush: 86 Begin loop 29 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.492846839Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.504626707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.504936992Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.504975707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.50498781Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505005778Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437} {"msg":"Crucible 06cb730e-0c00-4bed-9bc5-daef8804bd79 has session id: e6de3ae7-65c6-4f3c-9304-78cdd170986c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505357228Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Upstairs opts: Upstairs UUID: 06cb730e-0c00-4bed-9bc5-daef8804bd79, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505391526Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Crucible stats registered with UUID: 06cb730e-0c00-4bed-9bc5-daef8804bd79","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505410144Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:07.505423816Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505570244Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"io task","client":"2","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505605811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"io task","client":"0","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505621382Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"io task","client":"1","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505733501Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"task":"crutest"} {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505747822Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437} {"msg":"06cb730e-0c00-4bed-9bc5-daef8804bd79 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505769119Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505781961Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"client":"0","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505796182Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"client":"1","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505810223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"client":"2","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505824044Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"io task","client":"2","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505838495Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"io task","client":"0","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505852996Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"io task","client":"1","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"ds_connection connected from Ok(127.0.0.1:50017)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505930807Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"io task","client":"2","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"ds_connection connected from Ok(127.0.0.1:39878)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.505947177Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"io task","client":"1","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"ds_connection connected from Ok(127.0.0.1:52328)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.50605304Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"io task","client":"0","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.506716983Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"client":"2","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.506755279Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"client":"1","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.506804088Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"client":"0","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507076517Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507114143Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"[0]R flush_numbers[0..12]: [86, 86, 85, 75, 75, 82, 82, 80, 80, 85, 85, 82]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507176015Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"[0]R generation[0..12]: [57, 57, 56, 50, 50, 55, 55, 54, 54, 56, 56, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507193314Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507209643Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"[1]R flush_numbers[0..12]: [86, 86, 85, 75, 75, 82, 82, 80, 80, 85, 85, 82]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507224884Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"[1]R generation[0..12]: [57, 57, 56, 50, 50, 55, 55, 54, 54, 56, 56, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507240314Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507256784Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"[2]R flush_numbers[0..12]: [86, 86, 85, 75, 75, 82, 82, 80, 80, 85, 85, 82]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507272364Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"[2]R generation[0..12]: [57, 57, 56, 50, 50, 55, 55, 54, 54, 56, 56, 55]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507288084Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507307162Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507320884Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"Wait for a query_work_queue command to finish before sending IO downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507341231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Next flush: 87","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507354622Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507367794Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"":"downstairs","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507381795Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507394397Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"client":"0","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507408718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"client":"1","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Transition from WaitQuorum to Active","v":0,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 name":"crucible","level":30,"time":"2026-03-14T05:03:07.507425548Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"client":"2","":"downstairs","session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507440159Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.50745345Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"06cb730e-0c00-4bed-9bc5-daef8804bd79 is now active with session: e6de3ae7-65c6-4f3c-9304-78cdd170986c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507468451Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507481153Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437,"session_id":"e6de3ae7-65c6-4f3c-9304-78cdd170986c"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507494414Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437} {"msg":"Activated sub_volume 06cb730e-0c00-4bed-9bc5-daef8804bd79","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:07.507505897Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 517 len 24576 data: 4 3 4 3 2 2 02/30 Read block 381 len 16384 03/30 Read block 294 len 12288 04/30 Read block 244 len 16384 05/30 Read block 248 len 36864 06/30 Read block 74 len 32768 07/30 Write block 54 len 16384 data: 5 5 4 4 08/30 Flush 09/30 Flush 10/30 Write block 205 len 24576 data: 8 6 6 7 8 6 11/30 Write block 470 len 36864 data: 3 3 4 3 3 2 4 6 8 12/30 Write block 234 len 12288 data: 5 8 7 13/30 Write block 27 len 32768 data: 3 5 5 6 6 5 6 6 14/30 Read block 407 len 8192 15/30 Write block 572 len 4096 data: 3 16/30 Write block 592 len 8192 data: 5 5 17/30 Read block 72 len 36864 18/30 Read block 332 len 32768 19/30 Write block 28 len 24576 data: 6 6 7 7 6 7 20/30 Write block 33 len 28672 data: 8 7 8 8 6 4 5 21/30 Read block 195 len 24576 22/30 Flush 23/30 Write block 590 len 16384 data: 7 6 6 6 24/30 Read block 505 len 8192 25/30 Read block 323 len 16384 26/30 Read block 365 len 16384 27/30 Read block 461 len 40960 28/30 Read block 558 len 16384 29/30 Read block 501 len 8192 30/30 Read block 554 len 24576 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:21 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1009 Acked Write 6 Done Done Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Write 3 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Read 2 Done Sent Done false 1014 Acked Write 1 Done Sent Done false 1015 Acked Write 2 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Read 8 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 4 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Read 4 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 20 1 21 Done 21 1 20 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1008)) Some(JobId(1021)) Downstairs last five completed: 1008 1007 1006 1005 1004 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:07.519136878Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1437} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 86 86 86 F F F 1 020-039 58 57 58 89 86 89 F F F <--- 2 040-059 58 58 58 87 87 87 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 55 55 55 82 82 82 F F F 6 120-139 55 55 55 82 82 82 F F F 7 140-159 54 54 54 80 80 80 F F F 8 160-179 54 54 54 80 80 80 F F F 9 180-199 56 56 56 85 85 85 F F F 10 200-219 58 56 58 89 85 89 F T F <--- 11 220-239 58 55 58 89 82 89 F F F <--- 12 240-259 55 55 55 82 82 82 F F F 13 260-279 54 54 54 80 80 80 F F F 14 280-299 54 54 54 81 81 81 F F F 15 300-319 52 52 52 76 76 76 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 52 52 52 78 78 78 F F F 18 360-379 57 57 57 86 86 86 F F F 19 380-399 53 53 53 79 79 79 F F F 20 400-419 56 56 56 83 83 83 F F F 21 420-439 56 56 56 83 83 83 F F F 22 440-459 56 56 56 83 83 83 F F F 23 460-479 58 52 58 89 76 89 F F F <--- 24 480-499 56 56 56 83 83 83 F F F 25 500-519 58 58 58 87 87 87 F F F 26 520-539 58 58 58 87 87 87 F F F 27 540-559 56 56 56 83 83 83 F F F 28 560-579 58 56 58 89 83 89 F F F <--- 29 580-599 58 57 58 89 86 89 T F T <--- Max gen: 58, Max flush: 89 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 86 86 86 F F F 1 020-039 58 58 58 89 89 89 F F F 2 040-059 58 58 58 87 87 87 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 55 55 55 82 82 82 F F F 6 120-139 55 55 55 82 82 82 F F F 7 140-159 54 54 54 80 80 80 F F F 8 160-179 54 54 54 80 80 80 F F F 9 180-199 56 56 56 85 85 85 F F F 10 200-219 58 58 58 89 89 89 F F F 11 220-239 58 58 58 89 89 89 F F F 12 240-259 55 55 55 82 82 82 F F F 13 260-279 54 54 54 80 80 80 F F F 14 280-299 54 54 54 81 81 81 F F F 15 300-319 52 52 52 76 76 76 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 52 52 52 78 78 78 F F F 18 360-379 57 57 57 86 86 86 F F F 19 380-399 53 53 53 79 79 79 F F F 20 400-419 56 56 56 83 83 83 F F F 21 420-439 56 56 56 83 83 83 F F F 22 440-459 56 56 56 83 83 83 F F F 23 460-479 58 58 58 89 89 89 F F F 24 480-499 56 56 56 83 83 83 F F F 25 500-519 58 58 58 87 87 87 F F F 26 520-539 58 58 58 87 87 87 F F F 27 540-559 56 56 56 83 83 83 F F F 28 560-579 58 58 58 89 89 89 F F F 29 580-599 59 59 59 90 90 90 F F F Max gen: 59, Max flush: 90 Begin loop 30 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.821522239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:11.832125286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.83218162Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.842559609Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.842849596Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.842888002Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.842900375Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.842918343Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445} {"msg":"Crucible 609570e5-84e4-482f-861b-7d25e31da46b has session id: 5819c27f-5c3f-4a21-8c31-6904acd5cfe2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.84328913Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Upstairs opts: Upstairs UUID: 609570e5-84e4-482f-861b-7d25e31da46b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843323319Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Crucible stats registered with UUID: 609570e5-84e4-482f-861b-7d25e31da46b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843338979Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:11.84335251Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843506394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"io task","client":"2","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843541921Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"io task","client":"0","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843558071Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"io task","client":"1","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843657529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"task":"crutest"} {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.84367172Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445} {"msg":"609570e5-84e4-482f-861b-7d25e31da46b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843702081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843716072Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"client":"0","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843732512Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"client":"1","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843746413Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"client":"2","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843759974Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"io task","client":"2","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843774515Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"io task","client":"0","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843788796Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"io task","client":"1","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"ds_connection connected from Ok(127.0.0.1:53857)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843871944Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"io task","client":"2","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"ds_connection connected from Ok(127.0.0.1:47384)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843899697Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"io task","client":"0","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"ds_connection connected from Ok(127.0.0.1:42745)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.843967444Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"io task","client":"1","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.844628219Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"client":"2","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.844657191Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"client":"0","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.84467416Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"client":"1","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.844996447Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845041999Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"[0]R flush_numbers[0..12]: [86, 89, 87, 75, 75, 82, 82, 80, 80, 85, 89, 89]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845116722Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"[0]R generation[0..12]: [57, 58, 58, 50, 50, 55, 55, 54, 54, 56, 58, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845133351Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845153079Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"[1]R flush_numbers[0..12]: [86, 89, 87, 75, 75, 82, 82, 80, 80, 85, 89, 89]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845168639Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"[1]R generation[0..12]: [57, 58, 58, 50, 50, 55, 55, 54, 54, 56, 58, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.84518389Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845200449Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"[2]R flush_numbers[0..12]: [86, 89, 87, 75, 75, 82, 82, 80, 80, 85, 89, 89]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845215949Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"[2]R generation[0..12]: [57, 58, 58, 50, 50, 55, 55, 54, 54, 56, 58, 58]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.84523093Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.84524732Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Max found gen is 60","v":0,"name":"crucible","level":30,Wait for a query_work_queue command to finish before sending IO "time":"2026-03-14T05:03:11.845260721Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Generation requested: 60 >= found:60","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845280229Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Next flush: 91","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845293561Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845306912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"":"downstairs","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845321143Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845333935Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"client":"0","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845348156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"client":"1","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845362018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "client":"2","":"downstairs","session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845381925Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845395267Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"609570e5-84e4-482f-861b-7d25e31da46b is now active with session: 5819c27f-5c3f-4a21-8c31-6904acd5cfe2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845409927Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845422639Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445,"session_id":"5819c27f-5c3f-4a21-8c31-6904acd5cfe2"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845435981Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445} {"msg":"Activated sub_volume 609570e5-84e4-482f-861b-7d25e31da46b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:11.845447824Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 238 len 40960 data: 7 7 7 5 6 7 8 8 8 8 02/30 Read block 332 len 20480 03/30 Flush 04/30 Write block 2 len 36864 data: 4 6 5 6 5 5 4 4 4 05/30 Read block 354 len 24576 06/30 Read block 419 len 36864 07/30 Read block 139 len 16384 08/30 Read block 6 len 12288 09/30 Flush 10/30 Write block 45 len 40960 data: 4 2 2 3 4 4 4 6 6 6 11/30 Read block 587 len 12288 12/30 Read block 464 len 20480 13/30 Read block 9 len 20480 14/30 Read block 66 len 20480 15/30 Read block 40 len 40960 16/30 Read block 252 len 28672 17/30 Read block 77 len 16384 18/30 Read block 444 len 40960 19/30 Read block 158 len 12288 20/30 Flush 21/30 Write block 130 len 40960 data: 4 5 5 5 6 6 5 4 4 4 22/30 Write block 531 len 4096 data: 7 23/30 Read block 594 len 12288 24/30 Write block 173 len 40960 data: 3 3 3 4 5 5 4 3 3 5 25/30 Write block 110 len 16384 data: 7 6 6 6 26/30 Write block 307 len 24576 data: 9 12 10 10 9 11 27/30 Write block 556 len 20480 data: 9 10 11 10 10 28/30 Write block 345 len 40960 data: 5 6 6 5 5 6 5 6 5 3 29/30 Read block 158 len 4096 30/30 Write block 395 len 32768 data: 6 6 6 5 6 5 6 7 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 9 Done Done Done false 1004 Acked Read 6 Done Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Read 10 Sent Done Done false 1018 Acked Read 3 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 10 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Write 10 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 1 1 27 Done 2 26 26 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:11.858758889Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1445} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 60 60 86 92 92 T F F <--- 1 020-039 58 58 58 89 89 89 F F F 2 040-059 58 60 60 87 93 93 F F F <--- 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 55 55 55 82 82 82 F T T <--- 6 120-139 55 55 55 82 82 82 F T T <--- 7 140-159 54 54 54 80 80 80 F F F 8 160-179 54 54 54 80 80 80 F T T <--- 9 180-199 56 56 56 85 85 85 F T T <--- 10 200-219 58 58 58 89 89 89 F F F 11 220-239 60 60 60 91 91 91 F F F 12 240-259 60 60 60 91 91 91 F F F 13 260-279 54 54 54 80 80 80 F F F 14 280-299 54 54 54 81 81 81 F F F 15 300-319 52 52 52 76 76 76 F T T <--- 16 320-339 56 56 56 83 83 83 F F F 17 340-359 52 52 52 78 78 78 F T T <--- 18 360-379 57 57 57 86 86 86 F F F 19 380-399 53 53 53 79 79 79 F T T <--- 20 400-419 56 56 56 83 83 83 F T T <--- 21 420-439 56 56 56 83 83 83 F F F 22 440-459 56 56 56 83 83 83 F F F 23 460-479 58 58 58 89 89 89 F F F 24 480-499 56 56 56 83 83 83 F F F 25 500-519 58 58 58 87 87 87 F F F 26 520-539 58 58 58 87 87 87 F T T <--- 27 540-559 56 56 56 83 83 83 F T T <--- 28 560-579 58 58 58 89 89 89 F T T <--- 29 580-599 59 59 59 90 90 90 F F F Max gen: 60, Max flush: 93 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 92 92 92 F F F 1 020-039 58 58 58 89 89 89 F F F 2 040-059 60 60 60 93 93 93 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 61 61 61 94 94 94 F F F 6 120-139 61 61 61 94 94 94 F F F 7 140-159 54 54 54 80 80 80 F F F 8 160-179 61 61 61 94 94 94 F F F 9 180-199 61 61 61 94 94 94 F F F 10 200-219 58 58 58 89 89 89 F F F 11 220-239 60 60 60 91 91 91 F F F 12 240-259 60 60 60 91 91 91 F F F 13 260-279 54 54 54 80 80 80 F F F 14 280-299 54 54 54 81 81 81 F F F 15 300-319 61 61 61 94 94 94 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 61 61 61 94 94 94 F F F 18 360-379 57 57 57 86 86 86 F F F 19 380-399 61 61 61 94 94 94 F F F 20 400-419 61 61 61 94 94 94 F F F 21 420-439 56 56 56 83 83 83 F F F 22 440-459 56 56 56 83 83 83 F F F 23 460-479 58 58 58 89 89 89 F F F 24 480-499 56 56 56 83 83 83 F F F 25 500-519 58 58 58 87 87 87 F F F 26 520-539 61 61 61 94 94 94 F F F 27 540-559 61 61 61 94 94 94 F F F 28 560-579 61 61 61 94 94 94 F F F 29 580-599 59 59 59 90 90 90 F F F Max gen: 61, Max flush: 94 Begin loop 31 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.194438127Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:16.205035448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.205083018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.215318795Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.215611052Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.215649398Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.21566135Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.215679409Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453} {"msg":"Crucible 5be04c76-4c42-4c21-9d30-c046726be8d4 has session id: 7024214e-381e-4acf-8492-6a7c8f4f0f3c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216032687Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Upstairs opts: Upstairs UUID: 5be04c76-4c42-4c21-9d30-c046726be8d4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.21606014Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Crucible stats registered with UUID: 5be04c76-4c42-4c21-9d30-c046726be8d4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216077808Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:16.21609123Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216247052Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"io task","client":"2","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216270667Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"io task","client":"0","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216285208Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"io task","client":"1","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216403334Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"task":"crutest"} {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216418524Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453} {"msg":"5be04c76-4c42-4c21-9d30-c046726be8d4 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216435863Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216448496Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"client":"0","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216462087Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"client":"1","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216476348Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"client":"2","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216505859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"io task","client":"2","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216533342Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"io task","client":"0","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216576195Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"io task","client":"1","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"ds_connection connected from Ok(127.0.0.1:53479)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216628662Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"io task","client":"2","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"ds_connection connected from Ok(127.0.0.1:56354)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.216646181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"io task","client":"0","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"ds_connection connected from Ok(127.0.0.1:64679)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.21666378Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"io task","client":"1","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217327943Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"client":"2","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217363461Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"client":"1","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217498246Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"client":"0","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.21766672Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217704996Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"[0]R flush_numbers[0..12]: [92, 89, 93, 75, 75, 94, 94, 80, 94, 94, 89, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217842609Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"[0]R generation[0..12]: [60, 58, 60, 50, 50, 61, 61, 54, 61, 61, 58, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217859139Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217875739Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"[1]R flush_numbers[0..12]: [92, 89, 93, 75, 75, 94, 94, 80, 94, 94, 89, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217890689Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"[1]R generation[0..12]: [60, 58, 60, 50, 50, 61, 61, 54, 61, 61, 58, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.21790584Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217922109Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"[2]R flush_numbers[0..12]: [92, 89, 93, 75, 75, 94, 94, 80, 94, 94, 89, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.21793736Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"[2]R generation[0..12]: [60, 58, 60, 50, 50, 61, 61, 54, 61, 61, 58, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.2179526Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217972058Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.217985799ZWait for a query_work_queue command to finish before sending IO ","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.218005977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Next flush: 95","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.218019268Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.21803247Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"":"downstairs","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.218046641Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.218059103Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"client":"0","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.218073254Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"client":"1","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 level":30,"time":"2026-03-14T05:03:16.218090354Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"client":"2","":"downstairs","session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.218105134Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.218118766Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"5be04c76-4c42-4c21-9d30-c046726be8d4 is now active with session: 7024214e-381e-4acf-8492-6a7c8f4f0f3c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.218133427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.218146169Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453,"session_id":"7024214e-381e-4acf-8492-6a7c8f4f0f3c"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.2181593Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453} {"msg":"Activated sub_volume 5be04c76-4c42-4c21-9d30-c046726be8d4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:16.218171013Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 144 len 36864 data: 4 2 2 2 2 2 2 3 3 02/30 Write block 410 len 20480 data: 4 4 4 5 6 03/30 Read block 281 len 8192 04/30 Read block 510 len 8192 05/30 Read block 313 len 16384 06/30 Read block 348 len 8192 07/30 Read block 12 len 8192 08/30 Read block 361 len 36864 09/30 Read block 78 len 36864 10/30 Read block 361 len 12288 11/30 Read block 322 len 16384 12/30 Read block 317 len 28672 13/30 Write block 387 len 16384 data: 5 6 6 5 14/30 Write block 446 len 40960 data: 9 7 5 5 5 5 4 5 5 8 15/30 Write block 295 len 20480 data: 5 6 5 5 6 16/30 Read block 324 len 36864 17/30 Read block 97 len 28672 18/30 Flush 19/30 Write block 120 len 4096 data: 4 20/30 Read block 186 len 32768 21/30 Write block 523 len 36864 data: 2 2 3 4 6 6 6 7 8 22/30 Write block 247 len 20480 data: 9 10 8 8 8 23/30 Read block 421 len 24576 24/30 Read block 306 len 32768 25/30 Read block 89 len 24576 26/30 Read block 152 len 36864 27/30 Write block 176 len 16384 data: 5 6 6 5 28/30 Write block 199 len 24576 data: 5 5 6 6 7 6 29/30 Write block 207 len 8192 data: 7 8 30/30 Read block 292 len 32768 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Sent Done Done false 1001 Acked Write 5 Sent Done Done false 1002 Acked Read 2 Sent Done Done false 1003 Acked Read 2 Sent Done Done false 1004 Acked Read 4 Sent Done Done false 1005 Acked Read 2 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 3 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 8 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Write 2 Sent Done Done false 1029 Acked Read 8 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 1 31 Done 0 30 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:16.229885072Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1453} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 92 92 92 F F F 1 020-039 58 58 58 89 89 89 F F F 2 040-059 60 60 60 93 93 93 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 61 61 61 94 94 94 F F F 6 120-139 61 61 61 94 94 94 F T T <--- 7 140-159 54 62 62 80 95 95 F F F <--- 8 160-179 61 61 61 94 94 94 F T T <--- 9 180-199 61 61 61 94 94 94 F T T <--- 10 200-219 58 58 58 89 89 89 F T T <--- 11 220-239 60 60 60 91 91 91 F F F 12 240-259 60 60 60 91 91 91 F T T <--- 13 260-279 54 54 54 80 80 80 F F F 14 280-299 54 62 62 81 95 95 F F F <--- 15 300-319 61 61 61 94 94 94 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 61 61 61 94 94 94 F F F 18 360-379 57 57 57 86 86 86 F F F 19 380-399 61 62 62 94 95 95 F F F <--- 20 400-419 61 62 62 94 95 95 F F F <--- 21 420-439 56 56 56 83 83 83 F F F 22 440-459 56 62 62 83 95 95 F F F <--- 23 460-479 58 58 58 89 89 89 F F F 24 480-499 56 56 56 83 83 83 F F F 25 500-519 58 58 58 87 87 87 F F F 26 520-539 61 61 61 94 94 94 F T T <--- 27 540-559 61 61 61 94 94 94 F F F 28 560-579 61 61 61 94 94 94 F F F 29 580-599 59 59 59 90 90 90 F F F Max gen: 62, Max flush: 95 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 92 92 92 F F F 1 020-039 58 58 58 89 89 89 F F F 2 040-059 60 60 60 93 93 93 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 61 61 61 94 94 94 F F F 6 120-139 63 63 63 96 96 96 F F F 7 140-159 62 62 62 95 95 95 F F F 8 160-179 63 63 63 96 96 96 F F F 9 180-199 63 63 63 96 96 96 F F F 10 200-219 63 63 63 96 96 96 F F F 11 220-239 60 60 60 91 91 91 F F F 12 240-259 63 63 63 96 96 96 F F F 13 260-279 54 54 54 80 80 80 F F F 14 280-299 62 62 62 95 95 95 F F F 15 300-319 61 61 61 94 94 94 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 61 61 61 94 94 94 F F F 18 360-379 57 57 57 86 86 86 F F F 19 380-399 62 62 62 95 95 95 F F F 20 400-419 62 62 62 95 95 95 F F F 21 420-439 56 56 56 83 83 83 F F F 22 440-459 62 62 62 95 95 95 F F F 23 460-479 58 58 58 89 89 89 F F F 24 480-499 56 56 56 83 83 83 F F F 25 500-519 58 58 58 87 87 87 F F F 26 520-539 63 63 63 96 96 96 F F F 27 540-559 61 61 61 94 94 94 F F F 28 560-579 61 61 61 94 94 94 F F F 29 580-599 59 59 59 90 90 90 F F F Max gen: 63, Max flush: 96 Begin loop 32 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.555329032Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.567095467Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.567402554Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.567439291Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.567451464Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.567469263Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461} {"msg":"Crucible 21d50924-cfbd-4ad6-874b-6c886311e352 has session id: f20a8611-6b50-493b-99d0-6d4ab3f361f4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.567785554Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Upstairs opts: Upstairs UUID: 21d50924-cfbd-4ad6-874b-6c886311e352, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.567809779Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Crucible stats registered with UUID: 21d50924-cfbd-4ad6-874b-6c886311e352","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.567824259Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:20.567866263Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.56804701Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"io task","client":"2","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568081818Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"io task","client":"0","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568097808Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"io task","client":"1","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.56818899Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"task":"crutest"} {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568204121Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461} {"msg":"21d50924-cfbd-4ad6-874b-6c886311e352 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.56822136Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568233962Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"client":"0","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568251081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"client":"1","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568265352Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"client":"2","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568278984Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"io task","client":"2","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568293125Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"io task","client":"0","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568307476Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"io task","client":"1","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"ds_connection connected from Ok(127.0.0.1:57912)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568362062Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"io task","client":"2","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"ds_connection connected from Ok(127.0.0.1:53678)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568501614Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"io task","client":"1","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"ds_connection connected from Ok(127.0.0.1:55013)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.568529716Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"io task","client":"0","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569103316Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"client":"2","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569131188Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"client":"1","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569147898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"client":"0","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569453795Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569496389Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"[0]R flush_numbers[0..12]: [92, 89, 93, 75, 75, 94, 96, 95, 96, 96, 96, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.56955797Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"[0]R generation[0..12]: [60, 58, 60, 50, 50, 61, 63, 62, 63, 63, 63, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569574869Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569594157Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"[1]R flush_numbers[0..12]: [92, 89, 93, 75, 75, 94, 96, 95, 96, 96, 96, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569610077Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"[1]R generation[0..12]: [60, 58, 60, 50, 50, 61, 63, 62, 63, 63, 63, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569625548Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569641767Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"[2]R flush_numbers[0..12]: [92, 89, 93, 75, 75, 94, 96, 95, 96, 96, 96, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569657138Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"[2]R generation[0..12]: [60, 58, 60, 50, 50, 61, 63, 62, 63, 63, 63, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569672478Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569688868Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:03:20.569702259Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569723256Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Next flush: 97","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569736348Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569749749Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"":"downstairs","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569763631Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569776173Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"client":"0","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569790114Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"client":"1","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:03:20.569805804Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"client":"2","":"downstairs","session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569822434Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569836095Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"21d50924-cfbd-4ad6-874b-6c886311e352 is now active with session: f20a8611-6b50-493b-99d0-6d4ab3f361f4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569850996Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569863728Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461,"session_id":"f20a8611-6b50-493b-99d0-6d4ab3f361f4"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.56987687Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461} {"msg":"Activated sub_volume 21d50924-cfbd-4ad6-874b-6c886311e352","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:20.569888232Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 388 len 20480 data: 7 7 6 4 4 02/30 Read block 259 len 32768 03/30 Read block 292 len 36864 04/30 Write block 195 len 4096 data: 5 05/30 Read block 130 len 40960 06/30 Read block 324 len 24576 07/30 Write block 248 len 32768 data: 11 9 9 9 5 5 6 5 08/30 Read block 344 len 8192 09/30 Write block 481 len 20480 data: 8 7 6 5 4 10/30 Read block 452 len 8192 11/30 Write block 555 len 16384 data: 8 10 11 12 12/30 Read block 427 len 16384 13/30 Write block 208 len 40960 data: 9 9 7 6 7 9 7 8 10 10 14/30 Write block 146 len 12288 data: 3 3 3 15/30 Write block 386 len 16384 data: 4 6 8 8 16/30 Flush 17/30 Flush 18/30 Flush 19/30 Write block 146 len 20480 data: 4 4 4 3 3 20/30 Flush 21/30 Flush 22/30 Write block 582 len 12288 data: 9 7 7 23/30 Write block 492 len 20480 data: 5 4 4 3 2 24/30 Read block 157 len 4096 25/30 Write block 150 len 40960 data: 4 4 4 4 4 5 5 5 6 6 26/30 Write block 203 len 4096 data: 8 27/30 Read block 381 len 28672 28/30 Read block 149 len 12288 29/30 Read block 543 len 40960 30/30 Read block 543 len 12288 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 8 Done Sent Done false 1002 Acked Read 9 Done Sent Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Write 8 Done Sent Done false 1007 Acked Read 2 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Write 4 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Read 7 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 1 30 Done 30 1 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:20.58165111Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1461} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 92 92 92 F F F 1 020-039 58 58 58 89 89 89 F F F 2 040-059 60 60 60 93 93 93 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 61 61 61 94 94 94 F F F 6 120-139 63 63 63 96 96 96 F F F 7 140-159 64 62 64 100 95 100 T F T <--- 8 160-179 63 63 63 96 96 96 F F F 9 180-199 64 63 64 97 96 97 F F F <--- 10 200-219 64 63 64 97 96 97 T F T <--- 11 220-239 60 60 60 91 91 91 F F F 12 240-259 64 63 64 97 96 97 F F F <--- 13 260-279 54 54 54 80 80 80 F F F 14 280-299 62 62 62 95 95 95 F F F 15 300-319 61 61 61 94 94 94 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 61 61 61 94 94 94 F F F 18 360-379 57 57 57 86 86 86 F F F 19 380-399 64 62 64 97 95 97 F T F <--- 20 400-419 62 62 62 95 95 95 F F F 21 420-439 56 56 56 83 83 83 F F F 22 440-459 62 62 62 95 95 95 F F F 23 460-479 58 58 58 89 89 89 F F F 24 480-499 64 56 64 97 83 97 T F T <--- 25 500-519 58 58 58 87 87 87 F F F 26 520-539 63 63 63 96 96 96 F F F 27 540-559 64 61 64 97 94 97 F F F <--- 28 560-579 61 61 61 94 94 94 F F F 29 580-599 59 59 59 90 90 90 T F T <--- Max gen: 64, Max flush: 100 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 92 92 92 F F F 1 020-039 58 58 58 89 89 89 F F F 2 040-059 60 60 60 93 93 93 F F F 3 060-079 50 50 50 75 75 75 F F F 4 080-099 50 50 50 75 75 75 F F F 5 100-119 61 61 61 94 94 94 F F F 6 120-139 63 63 63 96 96 96 F F F 7 140-159 65 65 65 101 101 101 F F F 8 160-179 63 63 63 96 96 96 F F F 9 180-199 64 64 64 97 97 97 F F F 10 200-219 65 65 65 101 101 101 F F F 11 220-239 60 60 60 91 91 91 F F F 12 240-259 64 64 64 97 97 97 F F F 13 260-279 54 54 54 80 80 80 F F F 14 280-299 62 62 62 95 95 95 F F F 15 300-319 61 61 61 94 94 94 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 61 61 61 94 94 94 F F F 18 360-379 57 57 57 86 86 86 F F F 19 380-399 64 64 64 97 97 97 F F F 20 400-419 62 62 62 95 95 95 F F F 21 420-439 56 56 56 83 83 83 F F F 22 440-459 62 62 62 95 95 95 F F F 23 460-479 58 58 58 89 89 89 F F F 24 480-499 65 65 65 101 101 101 F F F 25 500-519 58 58 58 87 87 87 F F F 26 520-539 63 63 63 96 96 96 F F F 27 540-559 64 64 64 97 97 97 F F F 28 560-579 61 61 61 94 94 94 F F F 29 580-599 65 65 65 101 101 101 F F F Max gen: 65, Max flush: 101 Begin loop 33 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.904227402Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.915809294Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916132051Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916169248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.91618161Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916199649Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469} {"msg":"Crucible a57fadeb-c9ee-4700-8828-0a87575e2961 has session id: a0f31694-4ff2-4b03-9d94-8543ba8cca3b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916534958Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Upstairs opts: Upstairs UUID: a57fadeb-c9ee-4700-8828-0a87575e2961, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916565629Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Crucible stats registered with UUID: a57fadeb-c9ee-4700-8828-0a87575e2961","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.91658054Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:24.916596649Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916765873Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"io task","client":"2","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.91680196Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"io task","client":"0","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916817551Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"io task","client":"1","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916924773Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"task":"crutest"} {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916939644Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469} {"msg":"a57fadeb-c9ee-4700-8828-0a87575e2961 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916970345Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916983956Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"client":"0","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.916998197Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"client":"1","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.917012618Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"client":"2","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.917026439Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"io task","client":"2","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.917064226Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"io task","client":"0","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.917087051Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"io task","client":"1","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"ds_connection connected from Ok(127.0.0.1:41798)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.917107998Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"io task","client":"0","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"ds_connection connected from Ok(127.0.0.1:35297)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.917267008Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"io task","client":"2","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"ds_connection connected from Ok(127.0.0.1:37381)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.917289304Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"io task","client":"1","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.91783678Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"client":"0","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.917864922Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"client":"1","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.917977552Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"client":"2","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918206038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918224267Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"[0]R flush_numbers[0..12]: [92, 89, 93, 75, 75, 94, 96, 101, 96, 97, 101, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918374372Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"[0]R generation[0..12]: [60, 58, 60, 50, 50, 61, 63, 65, 63, 64, 65, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918391951Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918408751Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"[1]R flush_numbers[0..12]: [92, 89, 93, 75, 75, 94, 96, 101, 96, 97, 101, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918424361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"[1]R generation[0..12]: [60, 58, 60, 50, 50, 61, 63, 65, 63, 64, 65, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918439941Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918456151Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"[2]R flush_numbers[0..12]: [92, 89, 93, 75, 75, 94, 96, 101, 96, 97, 101, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918471871Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"[2]R generation[0..12]: [60, 58, 60, 50, 50, 61, 63, 65, 63, 64, 65, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918487711Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.91850496Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Max found gen is 66","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918522259Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918541827Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Next flush: 102","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918555239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.91856849Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"":"downstairs","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918582691Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918595383Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"client":"0","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918609365Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"client":"1","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918623376Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"client":"2","":"downstairs","session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:03:24.918637827Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918654506Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"a57fadeb-c9ee-4700-8828-0a87575e2961 is now active with session: a0f31694-4ff2-4b03-9d94-8543ba8cca3b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918669737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918682738Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469,"session_id":"a0f31694-4ff2-4b03-9d94-8543ba8cca3b"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.91869648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469} {"msg":"Activated sub_volume a57fadeb-c9ee-4700-8828-0a87575e2961","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:24.918708093Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 418 len 16384 data: 6 6 6 6 02/30 Read block 588 len 40960 03/30 Read block 125 len 32768 04/30 Read block 250 len 36864 05/30 Read block 203 len 36864 06/30 Flush 07/30 Write block 379 len 36864 data: 6 6 5 4 4 3 2 5 7 08/30 Write block 266 len 16384 data: 7 6 6 6 09/30 Read block 184 len 8192 10/30 Read block 164 len 8192 11/30 Flush 12/30 Write block 85 len 12288 data: 7 7 6 13/30 Read block 259 len 40960 14/30 Write block 387 len 32768 data: 8 9 9 7 5 5 5 5 15/30 Write block 580 len 40960 data: 9 9 10 8 8 6 5 5 5 7 16/30 Write block 79 len 20480 data: 8 6 6 5 6 17/30 Write block 405 len 16384 data: 5 5 7 6 18/30 Read block 42 len 4096 19/30 Read block 40 len 40960 20/30 Write block 389 len 24576 data: 10 8 6 6 6 6 21/30 Flush 22/30 Write block 577 len 40960 data: 8 8 7 10 10 11 9 9 7 6 23/30 Write block 308 len 8192 data: 13 11 24/30 Read block 277 len 28672 25/30 Write block 9 len 16384 data: 5 5 4 4 26/30 Read block 514 len 36864 27/30 Write block 212 len 16384 data: 8 10 8 9 28/30 Write block 17 len 24576 data: 4 5 5 5 5 5 29/30 Write block 490 len 32768 data: 6 6 6 5 5 4 3 2 30/30 Write block 194 len 8192 data: 5 6 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Sent false 1001 Acked Read 10 Done Done Sent false 1002 Acked Read 8 Done Done Sent false 1003 Acked Read 9 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 9 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Write 6 Sent Sent Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 30 36 Done 27 27 0 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:24.93085385Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1469} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 92 92 92 T T F <--- 1 020-039 58 58 58 89 89 89 T T F <--- 2 040-059 60 60 60 93 93 93 F F F 3 060-079 66 66 50 104 104 75 F F F <--- 4 080-099 66 66 50 104 104 75 F F F <--- 5 100-119 61 61 61 94 94 94 F F F 6 120-139 63 63 63 96 96 96 F F F 7 140-159 65 65 65 101 101 101 F F F 8 160-179 63 63 63 96 96 96 F F F 9 180-199 64 64 64 97 97 97 T T F <--- 10 200-219 65 65 65 101 101 101 T T F <--- 11 220-239 60 60 60 91 91 91 F F F 12 240-259 64 64 64 97 97 97 F F F 13 260-279 66 66 54 103 103 80 F F F <--- 14 280-299 62 62 62 95 95 95 F F F 15 300-319 61 61 61 94 94 94 T T F <--- 16 320-339 56 56 56 83 83 83 F F F 17 340-359 61 61 61 94 94 94 F F F 18 360-379 66 66 57 103 103 86 F F F <--- 19 380-399 66 66 64 104 104 97 F F F <--- 20 400-419 66 66 62 104 104 95 F F F <--- 21 420-439 66 66 56 102 102 83 F F F <--- 22 440-459 62 62 62 95 95 95 F F F 23 460-479 58 58 58 89 89 89 F F F 24 480-499 65 65 65 101 101 101 T T F <--- 25 500-519 58 58 58 87 87 87 F F F 26 520-539 63 63 63 96 96 96 F F F 27 540-559 64 64 64 97 97 97 F F F 28 560-579 61 61 61 94 94 94 T T F <--- 29 580-599 66 66 65 104 104 101 T T F <--- Max gen: 66, Max flush: 104 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 67 67 67 105 105 105 F F F 1 020-039 67 67 67 105 105 105 F F F 2 040-059 60 60 60 93 93 93 F F F 3 060-079 66 66 66 104 104 104 F F F 4 080-099 66 66 66 104 104 104 F F F 5 100-119 61 61 61 94 94 94 F F F 6 120-139 63 63 63 96 96 96 F F F 7 140-159 65 65 65 101 101 101 F F F 8 160-179 63 63 63 96 96 96 F F F 9 180-199 67 67 67 105 105 105 F F F 10 200-219 67 67 67 105 105 105 F F F 11 220-239 60 60 60 91 91 91 F F F 12 240-259 64 64 64 97 97 97 F F F 13 260-279 66 66 66 103 103 103 F F F 14 280-299 62 62 62 95 95 95 F F F 15 300-319 67 67 67 105 105 105 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 61 61 61 94 94 94 F F F 18 360-379 66 66 66 103 103 103 F F F 19 380-399 66 66 66 104 104 104 F F F 20 400-419 66 66 66 104 104 104 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 62 62 62 95 95 95 F F F 23 460-479 58 58 58 89 89 89 F F F 24 480-499 67 67 67 105 105 105 F F F 25 500-519 58 58 58 87 87 87 F F F 26 520-539 63 63 63 96 96 96 F F F 27 540-559 64 64 64 97 97 97 F F F 28 560-579 67 67 67 105 105 105 F F F 29 580-599 67 67 67 105 105 105 F F F Max gen: 67, Max flush: 105 Begin loop 34 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.256975044Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.268556246Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.268881002Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.268922266Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.268935018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.268952947Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477} {"msg":"Crucible 19166395-74bc-4357-992b-ebeefbfe827c has session id: 2838a29f-f956-4469-8baa-6d953d5abe51","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269315839Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Upstairs opts: Upstairs UUID: 19166395-74bc-4357-992b-ebeefbfe827c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269350227Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Crucible stats registered with UUID: 19166395-74bc-4357-992b-ebeefbfe827c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269365648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:29.269379669Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.26952151Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"io task","client":"2","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269545435Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"io task","client":"0","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269559935Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"io task","client":"1","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269684537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"task":"crutest"} {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269699818Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477} {"msg":"19166395-74bc-4357-992b-ebeefbfe827c active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269730238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.26974411Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"client":"0","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269760399Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"client":"1","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269774361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"client":"2","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269788052Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"io task","client":"2","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269802153Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"io task","client":"0","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269816244Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"io task","client":"1","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"ds_connection connected from Ok(127.0.0.1:40967)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.269906578Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"io task","client":"1","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"ds_connection connected from Ok(127.0.0.1:53608)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.270037625Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"io task","client":"0","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"ds_connection connected from Ok(127.0.0.1:36326)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.270065098Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"io task","client":"2","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.270588099Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"client":"1","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.270621498Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"client":"0","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.270638718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"client":"2","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.270940778Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.270986169Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"[0]R flush_numbers[0..12]: [105, 105, 93, 104, 104, 94, 96, 101, 96, 105, 105, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271023865Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"[0]R generation[0..12]: [67, 67, 60, 66, 66, 61, 63, 65, 63, 67, 67, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271040755Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271060303Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"[1]R flush_numbers[0..12]: [105, 105, 93, 104, 104, 94, 96, 101, 96, 105, 105, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271076183Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"[1]R generation[0..12]: [67, 67, 60, 66, 66, 61, 63, 65, 63, 67, 67, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271091443Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271107643Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"[2]R flush_numbers[0..12]: [105, 105, 93, 104, 104, 94, 96, 101, 96, 105, 105, 91]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271123143Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"[2]R generation[0..12]: [67, 67, 60, 66, 66, 61, 63, 65, 63, 67, 67, 60]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271138354Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271154324Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271167775Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271187453Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Next flush: 106","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271201254Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271214516Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"":"downstairs","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271228407Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271241209Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"client":"0","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.27125529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"client":"1","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271269231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"client":"2","":"downstairs","session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271285481Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271301961Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"19166395-74bc-4357-992b-ebeefbfe827c is now active with session: 2838a29f-f956-4469-8baa-6d953d5abe51","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271317281Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271329633Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477,"session_id":"2838a29f-f956-4469-8baa-6d953d5abe51"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271342925Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477} {"msg":"Activated sub_volume 19166395-74bc-4357-992b-ebeefbfe827c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:29.271354328Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 184 len 20480 data: 7 7 7 7 7 02/30 Write block 240 len 4096 data: 8 03/30 Read block 461 len 4096 04/30 Write block 250 len 40960 data: 10 10 6 6 7 6 7 8 8 8 05/30 Write block 513 len 24576 data: 6 6 8 5 5 4 06/30 Write block 52 len 40960 data: 7 7 7 6 5 5 4 5 7 7 07/30 Read block 472 len 32768 08/30 Read block 347 len 20480 09/30 Write block 14 len 8192 data: 5 6 10/30 Read block 283 len 4096 11/30 Read block 323 len 8192 12/30 Read block 366 len 24576 13/30 Flush 14/30 Write block 356 len 8192 data: 5 5 15/30 Read block 397 len 20480 16/30 Write block 592 len 20480 data: 7 7 4 4 4 17/30 Write block 451 len 36864 data: 6 5 6 6 9 8 7 7 8 18/30 Flush 19/30 Write block 310 len 16384 data: 11 10 12 11 20/30 Write block 113 len 8192 data: 7 5 21/30 Write block 227 len 4096 data: 5 22/30 Flush 23/30 Write block 161 len 12288 data: 4 3 2 24/30 Read block 135 len 12288 25/30 Flush 26/30 Write block 309 len 16384 data: 12 12 11 13 27/30 Write block 583 len 8192 data: 10 10 28/30 Read block 473 len 8192 29/30 Write block 390 len 8192 data: 9 7 30/30 Write block 226 len 12288 data: 4 6 5 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 10 Done Done Done false 1004 Acked Write 6 Done Done Sent false 1005 Acked Write 10 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Write 2 Done Done Sent false 1009 Acked Read 1 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 5 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 26 30 Done 28 28 4 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:29.281858907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1477} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 67 106 106 105 F F F <--- 1 020-039 67 67 67 105 105 105 F F F 2 040-059 68 68 60 106 106 93 F F F <--- 3 060-079 68 68 66 106 106 104 F F F <--- 4 080-099 66 66 66 104 104 104 F F F 5 100-119 68 68 61 108 108 94 F F F <--- 6 120-139 63 63 63 96 96 96 F F F 7 140-159 65 65 65 101 101 101 F F F 8 160-179 68 68 63 109 109 96 F F F <--- 9 180-199 68 68 67 106 106 105 F F T <--- 10 200-219 67 67 67 105 105 105 F F F 11 220-239 68 68 60 108 108 91 T T F <--- 12 240-259 68 68 64 106 106 97 F F T <--- 13 260-279 66 66 66 103 103 103 F F F 14 280-299 62 62 62 95 95 95 F F F 15 300-319 68 68 67 108 108 105 T T F <--- 16 320-339 56 56 56 83 83 83 F F F 17 340-359 68 68 61 107 107 94 F F F <--- 18 360-379 66 66 66 103 103 103 F F F 19 380-399 66 66 66 104 104 104 T T F <--- 20 400-419 66 66 66 104 104 104 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 68 68 62 107 107 95 F F F <--- 23 460-479 58 58 58 89 89 89 F F F 24 480-499 67 67 67 105 105 105 F F F 25 500-519 68 68 58 106 106 87 F F F <--- 26 520-539 63 63 63 96 96 96 F F F 27 540-559 64 64 64 97 97 97 F F F 28 560-579 67 67 67 105 105 105 F F F 29 580-599 68 68 67 107 107 105 T T F <--- Max gen: 68, Max flush: 109 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 68 68 68 106 106 106 F F F 1 020-039 67 67 67 105 105 105 F F F 2 040-059 68 68 68 106 106 106 F F F 3 060-079 68 68 68 106 106 106 F F F 4 080-099 66 66 66 104 104 104 F F F 5 100-119 68 68 68 108 108 108 F F F 6 120-139 63 63 63 96 96 96 F F F 7 140-159 65 65 65 101 101 101 F F F 8 160-179 68 68 68 109 109 109 F F F 9 180-199 68 68 68 106 106 106 F F F 10 200-219 67 67 67 105 105 105 F F F 11 220-239 69 69 69 110 110 110 F F F 12 240-259 68 68 68 106 106 106 F F F 13 260-279 66 66 66 103 103 103 F F F 14 280-299 62 62 62 95 95 95 F F F 15 300-319 69 69 69 110 110 110 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 68 68 68 107 107 107 F F F 18 360-379 66 66 66 103 103 103 F F F 19 380-399 69 69 69 110 110 110 F F F 20 400-419 66 66 66 104 104 104 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 68 68 68 107 107 107 F F F 23 460-479 58 58 58 89 89 89 F F F 24 480-499 67 67 67 105 105 105 F F F 25 500-519 68 68 68 106 106 106 F F F 26 520-539 63 63 63 96 96 96 F F F 27 540-559 64 64 64 97 97 97 F F F 28 560-579 67 67 67 105 105 105 F F F 29 580-599 69 69 69 110 110 110 F F F Max gen: 69, Max flush: 110 Begin loop 35 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.604788388Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.61636946Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.616682533Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.616722188Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.616738228Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.616755937Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485} {"msg":"Crucible 83ecf762-f45f-44ff-b09c-430bbfef1e64 has session id: 718e5c11-adde-4f09-a173-24135ef59da9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617089468Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Upstairs opts: Upstairs UUID: 83ecf762-f45f-44ff-b09c-430bbfef1e64, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617122837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Crucible stats registered with UUID: 83ecf762-f45f-44ff-b09c-430bbfef1e64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617140965Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:33.617154317Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617322691Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"io task","client":"1","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617346876Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"io task","client":"0","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617361787Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"io task","client":"2","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617473496Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"task":"crutest"} {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617490286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485} {"msg":"83ecf762-f45f-44ff-b09c-430bbfef1e64 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617507035Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617519807Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"client":"0","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617533948Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"client":"1","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.61754794Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"client":"2","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617561781Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"io task","client":"2","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617597658Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"io task","client":"0","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617622263Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"io task","client":"1","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"ds_connection connected from Ok(127.0.0.1:38414)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617638473Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"io task","client":"1","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"ds_connection connected from Ok(127.0.0.1:62890)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617766072Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"io task","client":"2","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"ds_connection connected from Ok(127.0.0.1:53240)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.617787639Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"io task","client":"0","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618275982Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"client":"2","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618296899Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"client":"0","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618316776Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"client":"1","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618641592Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618685864Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"[0]R flush_numbers[0..12]: [106, 105, 106, 106, 104, 108, 96, 101, 109, 106, 105, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618755181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"[0]R generation[0..12]: [68, 67, 68, 68, 66, 68, 63, 65, 68, 68, 67, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.61877182Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.61878845Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"[1]R flush_numbers[0..12]: [106, 105, 106, 106, 104, 108, 96, 101, 109, 106, 105, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.61880398Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"[1]R generation[0..12]: [68, 67, 68, 68, 66, 68, 63, 65, 68, 68, 67, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618819261Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.61883559Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"[2]R flush_numbers[0..12]: [106, 105, 106, 106, 104, 108, 96, 101, 109, 106, 105, 110]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618851181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"[2]R generation[0..12]: [68, 67, 68, 68, 66, 68, 63, 65, 68, 68, 67, 69]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618866341Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.61888467Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618898251Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618918238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Next flush: 111","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.61893158Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618944682Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"":"downstairs","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618959103Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618971825Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"client":"0","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.618986116Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"client":"1","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.619000657Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"client":"2","":"downstairs","session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:03:33.619015307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.619031597Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"83ecf762-f45f-44ff-b09c-430bbfef1e64 is now active with session: 718e5c11-adde-4f09-a173-24135ef59da9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.619046228Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.61905888Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485,"session_id":"718e5c11-adde-4f09-a173-24135ef59da9"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.619072521Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485} {"msg":"Activated sub_volume 83ecf762-f45f-44ff-b09c-430bbfef1e64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:33.619084354Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 101 len 12288 data: 4 5 6 03/30 Read block 160 len 28672 04/30 Read block 215 len 8192 05/30 Write block 231 len 8192 data: 3 4 06/30 Read block 228 len 24576 07/30 Write block 525 len 4096 data: 4 08/30 Read block 340 len 24576 09/30 Flush 10/30 Flush 11/30 Write block 361 len 28672 data: 6 6 4 4 4 7 9 12/30 Write block 468 len 8192 data: 3 3 13/30 Write block 291 len 4096 data: 8 14/30 Read block 499 len 24576 15/30 Write block 17 len 24576 data: 5 6 6 6 6 6 16/30 Read block 499 len 24576 17/30 Read block 493 len 16384 18/30 Read block 389 len 20480 19/30 Flush 20/30 Write block 269 len 8192 data: 7 8 21/30 Write block 195 len 12288 data: 7 3 4 22/30 Read block 89 len 8192 23/30 Read block 589 len 8192 24/30 Write block 509 len 12288 data: 7 6 5 25/30 Write block 352 len 4096 data: 7 26/30 Write block 499 len 4096 data: 3 27/30 Read block 387 len 20480 28/30 Read block 33 len 28672 29/30 Read block 403 len 36864 30/30 Read block 254 len 8192 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Write 2 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Write 2 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Read 2 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Read 2 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 1 28 Done 29 2 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1000)) Some(JobId(1018)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:33.62970407Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1485} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 70 68 70 114 106 114 F F F <--- 1 020-039 70 67 70 114 105 114 F F F <--- 2 040-059 68 68 68 106 106 106 F F F 3 060-079 68 68 68 106 106 106 F F F 4 080-099 66 66 66 104 104 104 F F F 5 100-119 70 68 70 112 108 112 F T F <--- 6 120-139 63 63 63 96 96 96 F F F 7 140-159 65 65 65 101 101 101 F F F 8 160-179 68 68 68 109 109 109 F F F 9 180-199 68 68 68 106 106 106 T F T <--- 10 200-219 67 67 67 105 105 105 F F F 11 220-239 70 69 70 112 110 112 F F F <--- 12 240-259 68 68 68 106 106 106 F F F 13 260-279 66 66 66 103 103 103 T F T <--- 14 280-299 70 62 70 114 95 114 F F F <--- 15 300-319 69 69 69 110 110 110 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 68 68 68 107 107 107 T F T <--- 18 360-379 70 66 70 114 103 114 F F F <--- 19 380-399 69 69 69 110 110 110 F F F 20 400-419 66 66 66 104 104 104 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 68 68 68 107 107 107 F F F 23 460-479 70 58 70 114 89 114 F F F <--- 24 480-499 67 67 67 105 105 105 T F T <--- 25 500-519 68 68 68 106 106 106 T F T <--- 26 520-539 70 63 70 112 96 112 F F F <--- 27 540-559 64 64 64 97 97 97 F F F 28 560-579 67 67 67 105 105 105 F F F 29 580-599 69 69 69 110 110 110 F F F Max gen: 70, Max flush: 114 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 70 70 70 114 114 114 F F F 1 020-039 70 70 70 114 114 114 F F F 2 040-059 68 68 68 106 106 106 F F F 3 060-079 68 68 68 106 106 106 F F F 4 080-099 66 66 66 104 104 104 F F F 5 100-119 70 70 70 112 112 112 F F F 6 120-139 63 63 63 96 96 96 F F F 7 140-159 65 65 65 101 101 101 F F F 8 160-179 68 68 68 109 109 109 F F F 9 180-199 71 71 71 115 115 115 F F F 10 200-219 67 67 67 105 105 105 F F F 11 220-239 70 70 70 112 112 112 F F F 12 240-259 68 68 68 106 106 106 F F F 13 260-279 71 71 71 115 115 115 F F F 14 280-299 70 70 70 114 114 114 F F F 15 300-319 69 69 69 110 110 110 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 71 71 71 115 115 115 F F F 18 360-379 70 70 70 114 114 114 F F F 19 380-399 69 69 69 110 110 110 F F F 20 400-419 66 66 66 104 104 104 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 68 68 68 107 107 107 F F F 23 460-479 70 70 70 114 114 114 F F F 24 480-499 71 71 71 115 115 115 F F F 25 500-519 71 71 71 115 115 115 F F F 26 520-539 70 70 70 112 112 112 F F F 27 540-559 64 64 64 97 97 97 F F F 28 560-579 67 67 67 105 105 105 F F F 29 580-599 69 69 69 110 110 110 F F F Max gen: 71, Max flush: 115 Begin loop 36 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.952606386Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:37.963217328Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.963272933Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.973663394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.973950034Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.973994406Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974006238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974023787Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493} {"msg":"Crucible 85b62829-8a48-44cb-91b1-668a18b13daa has session id: f2c17ef3-2032-4787-a8be-5ac952d5b7ef","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.97438496Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Upstairs opts: Upstairs UUID: 85b62829-8a48-44cb-91b1-668a18b13daa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974420158Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Crucible stats registered with UUID: 85b62829-8a48-44cb-91b1-668a18b13daa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974436068Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:37.97444961Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974587373Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"io task","client":"0","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974621492Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"io task","client":"2","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974636802Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"io task","client":"1","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974765821Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"task":"crutest"} {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974780172Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493} {"msg":"85b62829-8a48-44cb-91b1-668a18b13daa active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974799959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974812652Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"client":"0","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974826613Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"client":"1","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974840644Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"client":"2","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974854016Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"io task","client":"2","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974868257Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"io task","client":"0","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.974882648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"io task","client":"1","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"ds_connection connected from Ok(127.0.0.1:60154)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.975034172Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"io task","client":"2","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"ds_connection connected from Ok(127.0.0.1:34331)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.975068861Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"io task","client":"1","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"ds_connection connected from Ok(127.0.0.1:55606)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.9750854Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"io task","client":"0","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.975743167Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"client":"2","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.97576904Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"client":"1","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.97578556Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"client":"0","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976144395Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976161724Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"[0]R flush_numbers[0..12]: [114, 114, 106, 106, 104, 112, 96, 101, 109, 115, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976204667Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"[0]R generation[0..12]: [70, 70, 68, 68, 66, 70, 63, 65, 68, 71, 67, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976221696Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976238166Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"[1]R flush_numbers[0..12]: [114, 114, 106, 106, 104, 112, 96, 101, 109, 115, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976253806Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"[1]R generation[0..12]: [70, 70, 68, 68, 66, 70, 63, 65, 68, 71, 67, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976269126Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976285096Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"[2]R flush_numbers[0..12]: [114, 114, 106, 106, 104, 112, 96, 101, 109, 115, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976300856Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"[2]R generation[0..12]: [70, 70, 68, 68, 66, 70, 63, 65, 68, 71, 67, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976316107Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976332396Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976345888Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976365316Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Next flush: 116","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976378947Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976392499Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"":"downstairs","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976409518Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976423369Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"client":"0","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976439609Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"client":"1","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.97645342Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"client":"2","":"downstairs","session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"All required reconciliation work is completed"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.97647033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976483721Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"85b62829-8a48-44cb-91b1-668a18b13daa is now active with session: f2c17ef3-2032-4787-a8be-5ac952d5b7ef","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976498712Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976511654Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493,"session_id":"f2c17ef3-2032-4787-a8be-5ac952d5b7ef"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976525275Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493} {"msg":"Activated sub_volume 85b62829-8a48-44cb-91b1-668a18b13daa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:37.976536478Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 515 len 20480 data: 9 6 6 5 5 02/30 Write block 52 len 16384 data: 8 8 8 7 03/30 Read block 515 len 32768 04/30 Flush 05/30 Write block 592 len 8192 data: 8 8 06/30 Write block 348 len 8192 data: 6 6 07/30 Write block 360 len 20480 data: 6 7 7 5 5 08/30 Write block 76 len 4096 data: 8 09/30 Read block 438 len 36864 10/30 Read block 548 len 28672 11/30 Read block 20 len 4096 12/30 Write block 310 len 36864 data: 13 12 14 12 10 11 8 6 4 13/30 Write block 405 len 12288 data: 6 6 8 14/30 Read block 345 len 36864 15/30 Read block 448 len 32768 16/30 Write block 390 len 24576 data: 10 8 7 7 7 7 17/30 Read block 138 len 12288 18/30 Write block 506 len 24576 data: 6 6 7 8 7 6 19/30 Write block 129 len 12288 data: 2 5 6 20/30 Write block 490 len 28672 data: 7 7 7 6 6 5 4 21/30 Read block 385 len 24576 22/30 Write block 474 len 4096 data: 4 23/30 Write block 85 len 12288 data: 8 8 7 24/30 Read block 514 len 8192 25/30 Write block 292 len 32768 data: 8 6 5 6 7 6 6 7 26/30 Write block 272 len 16384 data: 6 5 7 7 27/30 Read block 266 len 32768 28/30 Read block 116 len 16384 29/30 Read block 152 len 8192 30/30 Read block 550 len 20480 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 2 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 9 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Write 8 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Read 5 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 0 27 Done 3 30 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:37.987756297Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1493} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 70 70 70 114 114 114 F F F 1 020-039 70 70 70 114 114 114 F F F 2 040-059 68 72 72 106 116 116 T F F <--- 3 060-079 68 68 68 106 106 106 F T T <--- 4 080-099 66 66 66 104 104 104 F T T <--- 5 100-119 70 70 70 112 112 112 F F F 6 120-139 63 63 63 96 96 96 F T T <--- 7 140-159 65 65 65 101 101 101 F F F 8 160-179 68 68 68 109 109 109 F F F 9 180-199 71 71 71 115 115 115 F F F 10 200-219 67 67 67 105 105 105 F F F 11 220-239 70 70 70 112 112 112 F F F 12 240-259 68 68 68 106 106 106 F F F 13 260-279 71 71 71 115 115 115 F T T <--- 14 280-299 70 70 70 114 114 114 F T T <--- 15 300-319 69 69 69 110 110 110 F T T <--- 16 320-339 56 56 56 83 83 83 F F F 17 340-359 71 71 71 115 115 115 F T T <--- 18 360-379 70 70 70 114 114 114 F T T <--- 19 380-399 69 69 69 110 110 110 F T T <--- 20 400-419 66 66 66 104 104 104 F T T <--- 21 420-439 66 66 66 102 102 102 F F F 22 440-459 68 68 68 107 107 107 F F F 23 460-479 70 70 70 114 114 114 F T T <--- 24 480-499 71 71 71 115 115 115 F T T <--- 25 500-519 71 72 72 115 116 116 T T T <--- 26 520-539 70 70 70 112 112 112 F F F 27 540-559 64 64 64 97 97 97 F F F 28 560-579 67 67 67 105 105 105 F F F 29 580-599 69 69 69 110 110 110 F T T <--- Max gen: 72, Max flush: 116 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 70 70 70 114 114 114 F F F 1 020-039 70 70 70 114 114 114 F F F 2 040-059 72 72 72 116 116 116 F F F 3 060-079 73 73 73 117 117 117 F F F 4 080-099 73 73 73 117 117 117 F F F 5 100-119 70 70 70 112 112 112 F F F 6 120-139 73 73 73 117 117 117 F F F 7 140-159 65 65 65 101 101 101 F F F 8 160-179 68 68 68 109 109 109 F F F 9 180-199 71 71 71 115 115 115 F F F 10 200-219 67 67 67 105 105 105 F F F 11 220-239 70 70 70 112 112 112 F F F 12 240-259 68 68 68 106 106 106 F F F 13 260-279 73 73 73 117 117 117 F F F 14 280-299 73 73 73 117 117 117 F F F 15 300-319 73 73 73 117 117 117 F F F 16 320-339 56 56 56 83 83 83 F F F 17 340-359 73 73 73 117 117 117 F F F 18 360-379 73 73 73 117 117 117 F F F 19 380-399 73 73 73 117 117 117 F F F 20 400-419 73 73 73 117 117 117 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 68 68 68 107 107 107 F F F 23 460-479 73 73 73 117 117 117 F F F 24 480-499 73 73 73 117 117 117 F F F 25 500-519 73 73 73 117 117 117 F F F 26 520-539 70 70 70 112 112 112 F F F 27 540-559 64 64 64 97 97 97 F F F 28 560-579 67 67 67 105 105 105 F F F 29 580-599 73 73 73 117 117 117 F F F Max gen: 73, Max flush: 117 Begin loop 37 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.3276753Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:42.338301962Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.338348703Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.348604318Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.348926706Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.348968239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.348980492Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.348998291Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501} {"msg":"Crucible 368d6640-f9ab-4575-8e58-9db55f3d5982 has session id: 6049a204-e530-441f-9317-2225ed377925","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349323256Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Upstairs opts: Upstairs UUID: 368d6640-f9ab-4575-8e58-9db55f3d5982, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349358634Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Crucible stats registered with UUID: 368d6640-f9ab-4575-8e58-9db55f3d5982","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349374584Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:42.349387936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349557199Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"io task","client":"1","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349580655Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"io task","client":"0","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349595595Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"io task","client":"2","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349729271Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"task":"crutest"} {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.34974676Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501} {"msg":"368d6640-f9ab-4575-8e58-9db55f3d5982 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349769576Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349782148Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"client":"0","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349795949Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"client":"1","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349809921Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"client":"2","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349823742Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"io task","client":"2","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349838473Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"io task","client":"0","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349855832Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"io task","client":"1","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"ds_connection connected from Ok(127.0.0.1:41400)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.349981823Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"io task","client":"0","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"ds_connection connected from Ok(127.0.0.1:61289)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.350118836Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"io task","client":"1","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"ds_connection connected from Ok(127.0.0.1:62705)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.350146729Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"io task","client":"2","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.35068648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"client":"1","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.350729623Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"client":"0","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.350783988Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"client":"2","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351091205Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351134678Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"[0]R flush_numbers[0..12]: [114, 114, 116, 117, 117, 112, 117, 101, 109, 115, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351192082Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"[0]R generation[0..12]: [70, 70, 72, 73, 73, 70, 73, 65, 68, 71, 67, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351213369Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351229968Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"[1]R flush_numbers[0..12]: [114, 114, 116, 117, 117, 112, 117, 101, 109, 115, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351245648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"[1]R generation[0..12]: [70, 70, 72, 73, 73, 70, 73, 65, 68, 71, 67, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351260789Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351280037Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"[2]R flush_numbers[0..12]: [114, 114, 116, 117, 117, 112, 117, 101, 109, 115, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351296067Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"[2]R generation[0..12]: [70, 70, 72, 73, 73, 70, 73, 65, 68, 71, 67, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351311417Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351327487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351341098Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351360976Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Next flush: 118","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351374517Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351387739Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"":"downstairs","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.3514022Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351415132Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"client":"0","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351428893Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"client":"1","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351442375Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"client":"2","":"downstairs","session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:03:42.351456756Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351470637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"368d6640-f9ab-4575-8e58-9db55f3d5982 is now active with session: 6049a204-e530-441f-9317-2225ed377925","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351485468Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351499938Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501,"session_id":"6049a204-e530-441f-9317-2225ed377925"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.35151324Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501} {"msg":"Activated sub_volume 368d6640-f9ab-4575-8e58-9db55f3d5982","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:42.351524513Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 285 len 40960 data: 5 4 5 6 7 6 9 9 7 6 02/30 Write block 505 len 36864 data: 6 7 7 8 9 8 7 8 7 03/30 Read block 583 len 24576 04/30 Write block 519 len 40960 data: 6 4 3 3 3 3 5 5 7 7 05/30 Read block 62 len 40960 06/30 Write block 571 len 8192 data: 4 4 07/30 Read block 507 len 16384 08/30 Write block 290 len 20480 data: 7 10 10 8 7 09/30 Write block 560 len 12288 data: 11 7 5 10/30 Write block 522 len 8192 data: 4 4 11/30 Read block 211 len 36864 12/30 Write block 176 len 24576 data: 6 7 7 6 4 4 13/30 Read block 200 len 12288 14/30 Read block 342 len 36864 15/30 Read block 316 len 16384 16/30 Read block 539 len 36864 17/30 Read block 341 len 16384 18/30 Write block 135 len 12288 data: 7 6 5 19/30 Read block 84 len 32768 20/30 Write block 338 len 16384 data: 5 4 3 3 21/30 Read block 521 len 4096 22/30 Write block 574 len 24576 data: 4 6 7 9 9 8 23/30 Write block 545 len 32768 data: 3 4 5 6 5 6 6 6 24/30 Read block 263 len 12288 25/30 Flush 26/30 Write block 19 len 8192 data: 7 7 27/30 Write block 139 len 32768 data: 5 4 4 4 4 5 3 5 28/30 Read block 292 len 24576 29/30 Read block 324 len 20480 30/30 Write block 335 len 32768 data: 4 5 5 6 5 4 4 5 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 9 Sent Done Done false 1002 Acked Read 6 Sent Done Done false 1003 Acked Write 10 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Write 3 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Read 9 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:42.36451432Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1501} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 70 70 70 114 114 114 F T T <--- 1 020-039 70 70 70 114 114 114 F T T <--- 2 040-059 72 72 72 116 116 116 F F F 3 060-079 73 73 73 117 117 117 F F F 4 080-099 73 73 73 117 117 117 F F F 5 100-119 70 70 70 112 112 112 F F F 6 120-139 73 74 74 117 118 118 F T T <--- 7 140-159 65 65 65 101 101 101 F T T <--- 8 160-179 68 74 74 109 118 118 F F F <--- 9 180-199 71 74 74 115 118 118 F F F <--- 10 200-219 67 67 67 105 105 105 F F F 11 220-239 70 70 70 112 112 112 F F F 12 240-259 68 68 68 106 106 106 F F F 13 260-279 73 73 73 117 117 117 F F F 14 280-299 73 74 74 117 118 118 T F F <--- 15 300-319 73 73 73 117 117 117 F F F 16 320-339 56 74 74 83 118 118 F T T <--- 17 340-359 73 74 74 117 118 118 F T T <--- 18 360-379 73 73 73 117 117 117 F F F 19 380-399 73 73 73 117 117 117 F F F 20 400-419 73 73 73 117 117 117 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 68 68 68 107 107 107 F F F 23 460-479 73 73 73 117 117 117 F F F 24 480-499 73 73 73 117 117 117 F F F 25 500-519 73 74 74 117 118 118 F F F <--- 26 520-539 70 74 74 112 118 118 F F F <--- 27 540-559 64 74 74 97 118 118 F F F <--- 28 560-579 67 74 74 105 118 118 F F F <--- 29 580-599 73 73 73 117 117 117 F F F Max gen: 74, Max flush: 118 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 72 72 72 116 116 116 F F F 3 060-079 73 73 73 117 117 117 F F F 4 080-099 73 73 73 117 117 117 F F F 5 100-119 70 70 70 112 112 112 F F F 6 120-139 75 75 75 119 119 119 F F F 7 140-159 75 75 75 119 119 119 F F F 8 160-179 74 74 74 118 118 118 F F F 9 180-199 74 74 74 118 118 118 F F F 10 200-219 67 67 67 105 105 105 F F F 11 220-239 70 70 70 112 112 112 F F F 12 240-259 68 68 68 106 106 106 F F F 13 260-279 73 73 73 117 117 117 F F F 14 280-299 74 74 74 118 118 118 F F F 15 300-319 73 73 73 117 117 117 F F F 16 320-339 75 75 75 119 119 119 F F F 17 340-359 75 75 75 119 119 119 F F F 18 360-379 73 73 73 117 117 117 F F F 19 380-399 73 73 73 117 117 117 F F F 20 400-419 73 73 73 117 117 117 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 68 68 68 107 107 107 F F F 23 460-479 73 73 73 117 117 117 F F F 24 480-499 73 73 73 117 117 117 F F F 25 500-519 74 74 74 118 118 118 F F F 26 520-539 74 74 74 118 118 118 F F F 27 540-559 74 74 74 118 118 118 F F F 28 560-579 74 74 74 118 118 118 F F F 29 580-599 73 73 73 117 117 117 F F F Max gen: 75, Max flush: 119 Begin loop 38 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.716483181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.728227101Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.728545481Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.728583017Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.72859528Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.728613228Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509} {"msg":"Crucible 1078e563-fc9e-4e0b-9d02-52cbbc4fc5c6 has session id: 92532144-1486-421d-b626-d6953bed2742","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.7289447Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Upstairs opts: Upstairs UUID: 1078e563-fc9e-4e0b-9d02-52cbbc4fc5c6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.728978979Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Crucible stats registered with UUID: 1078e563-fc9e-4e0b-9d02-52cbbc4fc5c6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.72900853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:46.729022471Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729162953Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"io task","client":"2","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729190226Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"io task","client":"0","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729204887Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"io task","client":"1","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729329089Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"task":"crutest"} {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729345568Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509} {"msg":"1078e563-fc9e-4e0b-9d02-52cbbc4fc5c6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729363167Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729375659Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"client":"0","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729389471Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"client":"1","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729403262Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"client":"2","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729417163Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"io task","client":"2","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729431474Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"io task","client":"0","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729445695Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"io task","client":"1","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"ds_connection connected from Ok(127.0.0.1:57774)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.72948597Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"io task","client":"1","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"ds_connection connected from Ok(127.0.0.1:50602)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.72950215Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"io task","client":"0","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"ds_connection connected from Ok(127.0.0.1:34017)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.729625912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"io task","client":"2","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.73023325Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"client":"1","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730253867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"client":"0","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730362509Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"client":"2","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730626094Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730651088Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"[0]R flush_numbers[0..12]: [119, 119, 116, 117, 117, 112, 119, 119, 118, 118, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730722223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"[0]R generation[0..12]: [75, 75, 72, 73, 73, 70, 75, 75, 74, 74, 67, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730740152Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730756811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"[1]R flush_numbers[0..12]: [119, 119, 116, 117, 117, 112, 119, 119, 118, 118, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730772791Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"[1]R generation[0..12]: [75, 75, 72, 73, 73, 70, 75, 75, 74, 74, 67, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730788352Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730804282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"[2]R flush_numbers[0..12]: [119, 119, 116, 117, 117, 112, 119, 119, 118, 118, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730819672Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"[2]R generation[0..12]: [75, 75, 72, 73, 73, 70, 75, 75, 74, 74, 67, 70]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730835092Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730851282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730865093Z","hostname"Wait for a query_work_queue command to finish before sending IO :"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730885341Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Next flush: 120","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.73090153Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730915112Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"":"downstairs","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730929193Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730941985Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"client":"0","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730956036Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"client":"1","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msgDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730972956Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"client":"2","":"downstairs","session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.730987307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.73101322Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"1078e563-fc9e-4e0b-9d02-52cbbc4fc5c6 is now active with session: 92532144-1486-421d-b626-d6953bed2742","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.731028181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.731040483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509,"session_id":"92532144-1486-421d-b626-d6953bed2742"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.731053775Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509} {"msg":"Activated sub_volume 1078e563-fc9e-4e0b-9d02-52cbbc4fc5c6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:46.731065537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 246 len 20480 data: 9 10 12 10 11 02/30 Flush 03/30 Write block 195 len 12288 data: 8 4 5 04/30 Write block 522 len 32768 data: 5 5 4 6 6 8 8 7 05/30 Write block 567 len 28672 data: 3 3 4 4 5 5 3 06/30 Read block 31 len 28672 07/30 Write block 236 len 20480 data: 8 6 8 8 9 08/30 Flush 09/30 Write block 192 len 20480 data: 5 5 6 9 5 10/30 Read block 71 len 36864 11/30 Flush 12/30 Write block 348 len 36864 data: 7 7 7 6 8 6 4 4 6 13/30 Flush 14/30 Flush 15/30 Write block 495 len 12288 data: 6 5 3 16/30 Read block 20 len 4096 17/30 Write block 349 len 12288 data: 8 8 7 18/30 Flush 19/30 Write block 357 len 40960 data: 6 5 4 7 8 8 6 6 5 8 20/30 Read block 248 len 8192 21/30 Read block 376 len 24576 22/30 Read block 333 len 16384 23/30 Read block 298 len 8192 24/30 Read block 292 len 4096 25/30 Write block 78 len 40960 data: 7 9 7 7 6 7 6 9 9 8 26/30 Write block 208 len 32768 data: 10 10 8 7 9 11 9 10 27/30 Read block 545 len 8192 28/30 Write block 503 len 16384 data: 6 6 7 8 29/30 Write block 134 len 16384 data: 7 8 7 6 30/30 Read block 573 len 36864 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Sent false 1001 Acked Flush 0 Done Done Sent false 1002 Acked Write 3 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Write 5 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 9 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 9 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) Some(JobId(1017)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:46.743261433Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1509} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 72 72 72 116 116 116 F F F 3 060-079 73 73 73 117 117 117 T T F <--- 4 080-099 73 73 73 117 117 117 T T F <--- 5 100-119 70 70 70 112 112 112 F F F 6 120-139 75 75 75 119 119 119 T T F <--- 7 140-159 75 75 75 119 119 119 F F F 8 160-179 74 74 74 118 118 118 F F F 9 180-199 76 76 74 122 122 118 F F F <--- 10 200-219 67 67 67 105 105 105 T T F <--- 11 220-239 76 76 70 121 121 112 F F F <--- 12 240-259 76 76 68 121 121 106 F F F <--- 13 260-279 73 73 73 117 117 117 F F F 14 280-299 74 74 74 118 118 118 F F F 15 300-319 73 73 73 117 117 117 F F F 16 320-339 75 75 75 119 119 119 F F F 17 340-359 76 76 75 125 125 119 T T F <--- 18 360-379 73 73 73 117 117 117 T T F <--- 19 380-399 73 73 73 117 117 117 F F F 20 400-419 73 73 73 117 117 117 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 68 68 68 107 107 107 F F F 23 460-479 73 73 73 117 117 117 F F F 24 480-499 76 76 73 125 125 117 F F F <--- 25 500-519 74 74 74 118 118 118 T T F <--- 26 520-539 76 76 74 121 121 118 F F F <--- 27 540-559 74 74 74 118 118 118 F F F 28 560-579 76 76 74 121 121 118 F F F <--- 29 580-599 73 73 73 117 117 117 F F F Max gen: 76, Max flush: 125 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 72 72 72 116 116 116 F F F 3 060-079 77 77 77 126 126 126 F F F 4 080-099 77 77 77 126 126 126 F F F 5 100-119 70 70 70 112 112 112 F F F 6 120-139 77 77 77 126 126 126 F F F 7 140-159 75 75 75 119 119 119 F F F 8 160-179 74 74 74 118 118 118 F F F 9 180-199 76 76 76 122 122 122 F F F 10 200-219 77 77 77 126 126 126 F F F 11 220-239 76 76 76 121 121 121 F F F 12 240-259 76 76 76 121 121 121 F F F 13 260-279 73 73 73 117 117 117 F F F 14 280-299 74 74 74 118 118 118 F F F 15 300-319 73 73 73 117 117 117 F F F 16 320-339 75 75 75 119 119 119 F F F 17 340-359 77 77 77 126 126 126 F F F 18 360-379 77 77 77 126 126 126 F F F 19 380-399 73 73 73 117 117 117 F F F 20 400-419 73 73 73 117 117 117 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 68 68 68 107 107 107 F F F 23 460-479 73 73 73 117 117 117 F F F 24 480-499 76 76 76 125 125 125 F F F 25 500-519 77 77 77 126 126 126 F F F 26 520-539 76 76 76 121 121 121 F F F 27 540-559 74 74 74 118 118 118 F F F 28 560-579 76 76 76 121 121 121 F F F 29 580-599 73 73 73 117 117 117 F F F Max gen: 77, Max flush: 126 Begin loop 39 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.061485289Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.073112123Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.073420939Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.073458155Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.073473576Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.073491425Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517} {"msg":"Crucible 35ad20c7-b8f5-49aa-9ba5-1d949a50b416 has session id: 4b1254e9-f7b7-4862-9e85-a07c2e593ea5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.073802449Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Upstairs opts: Upstairs UUID: 35ad20c7-b8f5-49aa-9ba5-1d949a50b416, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.073887036Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Crucible stats registered with UUID: 35ad20c7-b8f5-49aa-9ba5-1d949a50b416","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.073913639Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:51.07392799Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074075358Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"io task","client":"2","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074100262Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"io task","client":"1","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074115023Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"io task","client":"0","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074222585Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"task":"crutest"} {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074237985Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517} {"msg":"35ad20c7-b8f5-49aa-9ba5-1d949a50b416 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074254485Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074267527Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"client":"0","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074281578Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"client":"1","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074296039Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"client":"2","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074312139Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"io task","client":"2","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074327679Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"io task","client":"0","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.07434234Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"io task","client":"1","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"ds_connection connected from Ok(127.0.0.1:62460)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074360698Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"io task","client":"1","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"ds_connection connected from Ok(127.0.0.1:63189)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074375619Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"io task","client":"0","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"ds_connection connected from Ok(127.0.0.1:64485)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.074436251Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"io task","client":"2","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075098505Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"client":"1","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075150652Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"client":"0","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075219768Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"client":"2","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075490139Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075504879Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"[0]R flush_numbers[0..12]: [119, 119, 116, 126, 126, 112, 126, 119, 118, 122, 126, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075596921Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"[0]R generation[0..12]: [75, 75, 72, 77, 77, 70, 77, 75, 74, 76, 77, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.07561468Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.07563161Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"[1]R flush_numbers[0..12]: [119, 119, 116, 126, 126, 112, 126, 119, 118, 122, 126, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075648019Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"[1]R generation[0..12]: [75, 75, 72, 77, 77, 70, 77, 75, 74, 76, 77, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075666997Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075683967Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"[2]R flush_numbers[0..12]: [119, 119, 116, 126, 126, 112, 126, 119, 118, 122, 126, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075699897Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"[2]R generation[0..12]: [75, 75, 72, 77, 77, 70, 77, 75, 74, 76, 77, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075715187Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075731527Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Max found gen is 78","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:03:51.075745068Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075763827Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Next flush: 127","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075777138Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.07579068Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"":"downstairs","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075804891Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075817483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"client":"0","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075831644Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"client":"1","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075848693Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"client":"2","":"downstairs","session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075863454Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075878824Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"35ad20c7-b8f5-49aa-9ba5-1d949a50b416 is now active with session: 4b1254e9-f7b7-4862-9e85-a07c2e593ea5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075893735Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075906467Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517,"session_id":"4b1254e9-f7b7-4862-9e85-a07c2e593ea5"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075919519Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517} {"msg":"Activated sub_volume 35ad20c7-b8f5-49aa-9ba5-1d949a50b416","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:51.075931461Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 354 len 40960 data: 5 5 7 7 6 5 8 9 9 7 02/30 Write block 291 len 8192 data: 11 11 03/30 Read block 418 len 36864 04/30 Read block 589 len 12288 05/30 Read block 203 len 32768 06/30 Write block 253 len 12288 data: 7 8 7 07/30 Read block 350 len 12288 08/30 Write block 74 len 4096 data: 10 09/30 Write block 300 len 40960 data: 5 5 7 7 7 7 7 10 14 13 10/30 Write block 416 len 4096 data: 5 11/30 Read block 553 len 24576 12/30 Write block 383 len 32768 data: 5 4 3 6 9 10 11 11 13/30 Write block 252 len 20480 data: 7 8 9 8 8 14/30 Write block 59 len 8192 data: 6 8 15/30 Read block 569 len 40960 16/30 Read block 194 len 8192 17/30 Write block 491 len 20480 data: 8 8 7 7 7 18/30 Write block 443 len 32768 data: 10 10 10 10 8 6 6 6 19/30 Write block 185 len 4096 data: 8 20/30 Read block 302 len 4096 21/30 Write block 245 len 32768 data: 9 10 11 13 11 12 11 8 22/30 Write block 331 len 32768 data: 7 4 3 4 5 6 6 7 23/30 Read block 23 len 12288 24/30 Read block 443 len 12288 25/30 Flush 26/30 Write block 440 len 40960 data: 8 8 10 11 11 11 11 9 7 7 27/30 Read block 377 len 12288 28/30 Write block 163 len 32768 data: 3 2 2 2 3 3 3 3 29/30 Write block 116 len 16384 data: 3 4 3 4 30/30 Read block 456 len 24576 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Read 8 Done Done Done false 1005 Acked Write 3 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Write 1 Done Done Done false 1008 Acked Write 10 Done Done Done false 1009 Acked Write 1 Done Done Done false 1010 Acked Read 6 Done Done Done false 1011 Acked Write 8 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Write 1 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Write 8 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 6 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 19 20 Done 29 30 11 70 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:51.088119472Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1517} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 78 78 72 127 127 116 F F F <--- 3 060-079 78 78 77 127 127 126 F F T <--- 4 080-099 77 77 77 126 126 126 F F F 5 100-119 70 70 70 112 112 112 T T F <--- 6 120-139 77 77 77 126 126 126 F F F 7 140-159 75 75 75 119 119 119 F F F 8 160-179 74 74 74 118 118 118 T T F <--- 9 180-199 78 78 76 127 127 122 F F F <--- 10 200-219 77 77 77 126 126 126 F F F 11 220-239 76 76 76 121 121 121 F F F 12 240-259 78 78 76 127 127 121 F F T <--- 13 260-279 73 73 73 117 117 117 F F F 14 280-299 78 78 74 127 127 118 F F T <--- 15 300-319 78 78 73 127 127 117 F F T <--- 16 320-339 78 78 75 127 127 119 F F F <--- 17 340-359 78 78 77 127 127 126 F F T <--- 18 360-379 78 78 77 127 127 126 F F T <--- 19 380-399 78 78 73 127 127 117 F F F <--- 20 400-419 78 78 73 127 127 117 F F T <--- 21 420-439 66 66 66 102 102 102 F F F 22 440-459 78 78 68 127 127 107 T T F <--- 23 460-479 73 73 73 117 117 117 F F F 24 480-499 78 78 76 127 127 125 F F F <--- 25 500-519 77 77 77 126 126 126 F F F 26 520-539 76 76 76 121 121 121 F F F 27 540-559 74 74 74 118 118 118 F F F 28 560-579 76 76 76 121 121 121 F F F 29 580-599 73 73 73 117 117 117 F F F Max gen: 78, Max flush: 127 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 78 78 78 127 127 127 F F F 3 060-079 78 78 78 127 127 127 F F F 4 080-099 77 77 77 126 126 126 F F F 5 100-119 79 79 79 128 128 128 F F F 6 120-139 77 77 77 126 126 126 F F F 7 140-159 75 75 75 119 119 119 F F F 8 160-179 79 79 79 128 128 128 F F F 9 180-199 78 78 78 127 127 127 F F F 10 200-219 77 77 77 126 126 126 F F F 11 220-239 76 76 76 121 121 121 F F F 12 240-259 78 78 78 127 127 127 F F F 13 260-279 73 73 73 117 117 117 F F F 14 280-299 78 78 78 127 127 127 F F F 15 300-319 78 78 78 127 127 127 F F F 16 320-339 78 78 78 127 127 127 F F F 17 340-359 78 78 78 127 127 127 F F F 18 360-379 78 78 78 127 127 127 F F F 19 380-399 78 78 78 127 127 127 F F F 20 400-419 78 78 78 127 127 127 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 79 79 79 128 128 128 F F F 23 460-479 73 73 73 117 117 117 F F F 24 480-499 78 78 78 127 127 127 F F F 25 500-519 77 77 77 126 126 126 F F F 26 520-539 76 76 76 121 121 121 F F F 27 540-559 74 74 74 118 118 118 F F F 28 560-579 76 76 76 121 121 121 F F F 29 580-599 73 73 73 117 117 117 F F F Max gen: 79, Max flush: 128 Begin loop 40 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.412091496Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.423709126Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424035101Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424073337Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424085469Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424103218Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525} {"msg":"Crucible 7a9557e6-ad14-4bcb-a16a-81445793ec53 has session id: ef947c86-a401-4474-9f61-07129df5ed7f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424463292Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Upstairs opts: Upstairs UUID: 7a9557e6-ad14-4bcb-a16a-81445793ec53, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424489465Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Crucible stats registered with UUID: 7a9557e6-ad14-4bcb-a16a-81445793ec53","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424504326Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:55.424519926Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424683863Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"io task","client":"2","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424718631Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"io task","client":"1","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424734101Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"io task","client":"0","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424840615Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"task":"crutest"} {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424855095Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525} {"msg":"7a9557e6-ad14-4bcb-a16a-81445793ec53 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424885286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424899508Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"client":"0","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424913629Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"client":"1","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.42492731Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"client":"2","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424941001Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"io task","client":"2","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424955213Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"io task","client":"0","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.424969384Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"io task","client":"1","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"ds_connection connected from Ok(127.0.0.1:65223)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.425030225Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"io task","client":"2","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"ds_connection connected from Ok(127.0.0.1:42735)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.425057138Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"io task","client":"1","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"ds_connection connected from Ok(127.0.0.1:33417)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.425072029Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"io task","client":"0","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.425767782Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"client":"0","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.425799172Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"client":"1","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.425816731Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"client":"2","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426155898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426171428Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"[0]R flush_numbers[0..12]: [119, 119, 127, 127, 126, 128, 126, 119, 128, 127, 126, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426224425Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"[0]R generation[0..12]: [75, 75, 78, 78, 77, 79, 77, 75, 79, 78, 77, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426241334Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426257614Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"[1]R flush_numbers[0..12]: [119, 119, 127, 127, 126, 128, 126, 119, 128, 127, 126, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426273384Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"[1]R generation[0..12]: [75, 75, 78, 78, 77, 79, 77, 75, 79, 78, 77, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426288804Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426305034Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"[2]R flush_numbers[0..12]: [119, 119, 127, 127, 126, 128, 126, 119, 128, 127, 126, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426320624Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"[2]R generation[0..12]: [75, 75, 78, 78, 77, 79, 77, 75, 79, 78, 77, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426336375Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426352514Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426368594Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426382546Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs",Wait for a query_work_queue command to finish before sending IO "":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Next flush: 129","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426402103Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426415605Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"":"downstairs","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426429446Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426442278Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"client":"0","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426456449Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"client":"1","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426470171Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"client":"2","":"downstairs","session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:03:55.426484292Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426499402Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"7a9557e6-ad14-4bcb-a16a-81445793ec53 is now active with session: ef947c86-a401-4474-9f61-07129df5ed7f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426514643Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426527395Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525,"session_id":"ef947c86-a401-4474-9f61-07129df5ed7f"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426540666Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525} {"msg":"Activated sub_volume 7a9557e6-ad14-4bcb-a16a-81445793ec53","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:55.426552279Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 290 len 16384 data: 8 12 12 9 02/30 Read block 166 len 36864 03/30 Write block 165 len 4096 data: 3 04/30 Read block 59 len 36864 05/30 Flush 06/30 Write block 250 len 16384 data: 13 12 9 9 07/30 Read block 64 len 40960 08/30 Write block 350 len 20480 data: 9 8 9 7 6 09/30 Flush 10/30 Write block 216 len 8192 data: 11 11 11/30 Read block 256 len 32768 12/30 Write block 575 len 36864 data: 7 8 10 10 9 11 11 12 11 13/30 Write block 47 len 4096 data: 3 14/30 Write block 464 len 40960 data: 3 3 3 3 4 4 4 4 5 4 15/30 Flush 16/30 Write block 518 len 16384 data: 6 7 5 4 17/30 Read block 246 len 4096 18/30 Read block 190 len 36864 19/30 Write block 334 len 40960 data: 5 6 7 7 8 6 5 5 6 4 20/30 Write block 396 len 32768 data: 7 7 6 7 6 7 8 5 21/30 Read block 377 len 32768 22/30 Write block 106 len 28672 data: 6 5 5 7 8 7 7 23/30 Write block 533 len 32768 data: 7 7 5 4 3 2 3 5 24/30 Read block 57 len 32768 25/30 Read block 417 len 24576 26/30 Read block 276 len 8192 27/30 Read block 209 len 20480 28/30 Write block 203 len 20480 data: 9 7 9 7 8 29/30 Write block 492 len 8192 data: 9 8 30/30 Read block 56 len 20480 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:21 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1009 Acked Write 2 Done Done Done false 1010 Acked Read 8 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Write 1 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Write 2 Done Done Sent false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 20 21 Done 21 20 1 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1014)) Some(JobId(1008)) Downstairs last five completed: 1008 1007 1006 1005 1004 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:55.438642501Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1525} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 80 80 78 131 131 127 F F F <--- 3 060-079 78 78 78 127 127 127 F F F 4 080-099 77 77 77 126 126 126 F F F 5 100-119 79 79 79 128 128 128 T T F <--- 6 120-139 77 77 77 126 126 126 F F F 7 140-159 75 75 75 119 119 119 F F F 8 160-179 80 80 80 129 129 129 F F F 9 180-199 78 78 78 127 127 127 F F F 10 200-219 80 80 77 131 131 126 T T T <--- 11 220-239 76 76 76 121 121 121 F F F 12 240-259 80 80 80 130 130 130 F F F 13 260-279 73 73 73 117 117 117 F F F 14 280-299 80 80 80 129 129 129 F F F 15 300-319 78 78 78 127 127 127 F F F 16 320-339 78 78 78 127 127 127 T T F <--- 17 340-359 80 80 80 130 130 130 T T F <--- 18 360-379 78 78 78 127 127 127 F F F 19 380-399 78 78 78 127 127 127 T T F <--- 20 400-419 78 78 78 127 127 127 T T F <--- 21 420-439 66 66 66 102 102 102 F F F 22 440-459 79 79 79 128 128 128 F F F 23 460-479 80 80 73 131 131 117 F F F <--- 24 480-499 78 78 78 127 127 127 T T F <--- 25 500-519 77 77 77 126 126 126 T T F <--- 26 520-539 76 76 76 121 121 121 T T F <--- 27 540-559 74 74 74 118 118 118 T T F <--- 28 560-579 80 80 76 131 131 121 F F F <--- 29 580-599 80 80 73 131 131 117 F F F <--- Max gen: 80, Max flush: 131 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 80 80 80 131 131 131 F F F 3 060-079 78 78 78 127 127 127 F F F 4 080-099 77 77 77 126 126 126 F F F 5 100-119 81 81 81 132 132 132 F F F 6 120-139 77 77 77 126 126 126 F F F 7 140-159 75 75 75 119 119 119 F F F 8 160-179 80 80 80 129 129 129 F F F 9 180-199 78 78 78 127 127 127 F F F 10 200-219 81 81 81 132 132 132 F F F 11 220-239 76 76 76 121 121 121 F F F 12 240-259 80 80 80 130 130 130 F F F 13 260-279 73 73 73 117 117 117 F F F 14 280-299 80 80 80 129 129 129 F F F 15 300-319 78 78 78 127 127 127 F F F 16 320-339 81 81 81 132 132 132 F F F 17 340-359 81 81 81 132 132 132 F F F 18 360-379 78 78 78 127 127 127 F F F 19 380-399 81 81 81 132 132 132 F F F 20 400-419 81 81 81 132 132 132 F F F 21 420-439 66 66 66 102 102 102 F F F 22 440-459 79 79 79 128 128 128 F F F 23 460-479 80 80 80 131 131 131 F F F 24 480-499 81 81 81 132 132 132 F F F 25 500-519 81 81 81 132 132 132 F F F 26 520-539 81 81 81 132 132 132 F F F 27 540-559 81 81 81 132 132 132 F F F 28 560-579 80 80 80 131 131 131 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 81, Max flush: 132 Begin loop 41 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.764483582Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.776042348Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.776367314Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.776408188Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.776422739Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.776440778Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533} {"msg":"Crucible 7740e35f-165e-4b83-ad51-c50947bbf9cd has session id: 90d1b039-53cf-4d33-9ad6-99fc91f02a0e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.77677081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Upstairs opts: Upstairs UUID: 7740e35f-165e-4b83-ad51-c50947bbf9cd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.776806478Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Crucible stats registered with UUID: 7740e35f-165e-4b83-ad51-c50947bbf9cd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.776822118Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:59.776835759Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.776973313Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"io task","client":"2","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777008961Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"io task","client":"0","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777024821Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"io task","client":"1","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777127676Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"task":"crutest"} {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777142177Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533} {"msg":"7740e35f-165e-4b83-ad51-c50947bbf9cd active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777161835Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777174716Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"client":"0","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777188858Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"client":"1","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777202929Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"client":"2","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777218859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"io task","client":"2","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.77723313Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"io task","client":"0","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777247751Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"io task","client":"1","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"ds_connection connected from Ok(127.0.0.1:55658)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777284178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"io task","client":"1","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"ds_connection connected from Ok(127.0.0.1:47816)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777299728Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"io task","client":"2","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"ds_connection connected from Ok(127.0.0.1:39312)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777404722Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"io task","client":"0","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.777983038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"client":"1","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778017787Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"client":"0","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778195615Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"client":"2","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778336047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778373073Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"[0]R flush_numbers[0..12]: [119, 119, 131, 127, 126, 132, 126, 119, 129, 127, 132, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778572418Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"[0]R generation[0..12]: [75, 75, 80, 78, 77, 81, 77, 75, 80, 78, 81, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778589367Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778605877Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"[1]R flush_numbers[0..12]: [119, 119, 131, 127, 126, 132, 126, 119, 129, 127, 132, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778621697Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"[1]R generation[0..12]: [75, 75, 80, 78, 77, 81, 77, 75, 80, 78, 81, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778639836Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778656375Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"[2]R flush_numbers[0..12]: [119, 119, 131, 127, 126, 132, 126, 119, 129, 127, 132, 121]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778672145Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"[2]R generation[0..12]: [75, 75, 80, 78, 77, 81, 77, 75, 80, 78, 81, 76]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778687646Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778703596Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:03:59.778716837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778736395Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Next flush: 133","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778749777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778763108Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"":"downstairs","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778777659Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778790431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"client":"0","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778804702Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"client":"1","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778821272Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"client":"2","":"downstairs","session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778835783Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778850823Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"7740e35f-165e-4b83-ad51-c50947bbf9cd is now active with session: 90d1b039-53cf-4d33-9ad6-99fc91f02a0e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778865904Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778878586Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533,"session_id":"90d1b039-53cf-4d33-9ad6-99fc91f02a0e"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.778891877Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533} {"msg":"Activated sub_volume 7740e35f-165e-4b83-ad51-c50947bbf9cd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:03:59.77890318Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 429 len 36864 data: 3 3 4 6 6 8 8 8 8 02/30 Write block 155 len 16384 data: 6 6 6 7 03/30 Read block 466 len 40960 04/30 Read block 389 len 4096 05/30 Read block 429 len 40960 06/30 Read block 542 len 40960 07/30 Read block 310 len 36864 08/30 Flush 09/30 Write block 247 len 40960 data: 12 14 12 14 13 10 10 10 9 9 10/30 Flush 11/30 Write block 375 len 16384 data: 4 4 4 5 12/30 Write block 481 len 32768 data: 9 8 7 6 5 3 6 6 13/30 Write block 84 len 40960 data: 7 10 10 9 6 6 8 8 7 6 14/30 Write block 506 len 4096 data: 9 15/30 Flush 16/30 Write block 474 len 20480 data: 5 3 5 7 9 17/30 Read block 355 len 8192 18/30 Read block 553 len 4096 19/30 Write block 187 len 24576 data: 8 8 6 6 6 6 20/30 Read block 378 len 20480 21/30 Write block 566 len 16384 data: 4 4 4 5 22/30 Write block 279 len 16384 data: 5 4 3 4 23/30 Read block 179 len 16384 24/30 Write block 223 len 24576 data: 7 8 6 5 7 6 25/30 Read block 89 len 4096 26/30 Write block 547 len 16384 data: 6 7 6 7 27/30 Read block 479 len 8192 28/30 Read block 368 len 20480 29/30 Write block 203 len 16384 data: 10 8 10 8 30/30 Read block 221 len 40960 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 10 Done Done Sent false 1003 Acked Read 1 Done Done Sent false 1004 Acked Read 10 Done Done Sent false 1005 Acked Read 10 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 10 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Write 4 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Read 4 Done Done Sent false 1023 Acked Write 6 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 10 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 28 28 Done 30 30 2 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1014)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:03:59.791436224Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1533} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 80 80 80 131 131 131 F F F 3 060-079 78 78 78 127 127 127 F F F 4 080-099 82 82 77 135 135 126 F F F <--- 5 100-119 81 81 81 132 132 132 F F F 6 120-139 77 77 77 126 126 126 F F F 7 140-159 82 82 75 133 133 119 F F T <--- 8 160-179 80 80 80 129 129 129 F F F 9 180-199 78 78 78 127 127 127 T T F <--- 10 200-219 81 81 81 132 132 132 T T F <--- 11 220-239 76 76 76 121 121 121 T T F <--- 12 240-259 82 82 80 134 134 130 F F F <--- 13 260-279 73 73 73 117 117 117 T T F <--- 14 280-299 80 80 80 129 129 129 T T F <--- 15 300-319 78 78 78 127 127 127 F F F 16 320-339 81 81 81 132 132 132 F F F 17 340-359 81 81 81 132 132 132 F F F 18 360-379 82 82 78 135 135 127 F F F <--- 19 380-399 81 81 81 132 132 132 F F F 20 400-419 81 81 81 132 132 132 F F F 21 420-439 82 82 66 133 133 102 F F T <--- 22 440-459 79 79 79 128 128 128 F F F 23 460-479 80 80 80 131 131 131 T T F <--- 24 480-499 82 82 81 135 135 132 F F F <--- 25 500-519 82 82 81 135 135 132 F F F <--- 26 520-539 81 81 81 132 132 132 F F F 27 540-559 81 81 81 132 132 132 T T F <--- 28 560-579 80 80 80 131 131 131 T T F <--- 29 580-599 80 80 80 131 131 131 F F F Max gen: 82, Max flush: 135 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 80 80 80 131 131 131 F F F 3 060-079 78 78 78 127 127 127 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 81 81 81 132 132 132 F F F 6 120-139 77 77 77 126 126 126 F F F 7 140-159 82 82 82 133 133 133 F F F 8 160-179 80 80 80 129 129 129 F F F 9 180-199 83 83 83 136 136 136 F F F 10 200-219 83 83 83 136 136 136 F F F 11 220-239 83 83 83 136 136 136 F F F 12 240-259 82 82 82 134 134 134 F F F 13 260-279 83 83 83 136 136 136 F F F 14 280-299 83 83 83 136 136 136 F F F 15 300-319 78 78 78 127 127 127 F F F 16 320-339 81 81 81 132 132 132 F F F 17 340-359 81 81 81 132 132 132 F F F 18 360-379 82 82 82 135 135 135 F F F 19 380-399 81 81 81 132 132 132 F F F 20 400-419 81 81 81 132 132 132 F F F 21 420-439 82 82 82 133 133 133 F F F 22 440-459 79 79 79 128 128 128 F F F 23 460-479 83 83 83 136 136 136 F F F 24 480-499 82 82 82 135 135 135 F F F 25 500-519 82 82 82 135 135 135 F F F 26 520-539 81 81 81 132 132 132 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 83 83 83 136 136 136 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 83, Max flush: 136 Begin loop 42 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.120061565Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.131639409Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.131961647Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132011585Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132024127Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132041806Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541} {"msg":"Crucible ac8e87af-5bc0-4f87-8220-5dbf0e136dfd has session id: a3c7fc68-1698-4fdb-8db0-cced6ed986b8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132382033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Upstairs opts: Upstairs UUID: ac8e87af-5bc0-4f87-8220-5dbf0e136dfd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132417001Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Crucible stats registered with UUID: ac8e87af-5bc0-4f87-8220-5dbf0e136dfd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13243345Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:04.132448231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132615846Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"io task","client":"2","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13264057Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"io task","client":"0","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132655231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"io task","client":"1","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132780892Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"task":"crutest"} {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132795313Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541} {"msg":"ac8e87af-5bc0-4f87-8220-5dbf0e136dfd active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13281607Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132829142Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"client":"0","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132843123Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"client":"1","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132856904Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"client":"2","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132870546Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"io task","client":"2","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132884467Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"io task","client":"0","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132898628Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"io task","client":"1","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"ds_connection connected from Ok(127.0.0.1:33652)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.132962678Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"io task","client":"0","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"ds_connection connected from Ok(127.0.0.1:44411)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.133062275Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"io task","client":"2","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"ds_connection connected from Ok(127.0.0.1:64438)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.133124196Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"io task","client":"1","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.133665776Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"client":"0","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.133729266Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"client":"2","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.133841046Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"client":"1","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134016705Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134080565Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"[0]R flush_numbers[0..12]: [119, 119, 131, 127, 135, 132, 126, 133, 129, 136, 136, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134198571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"[0]R generation[0..12]: [75, 75, 80, 78, 82, 81, 77, 82, 80, 83, 83, 83]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13421546Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13423204Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"[1]R flush_numbers[0..12]: [119, 119, 131, 127, 135, 132, 126, 133, 129, 136, 136, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"[1]R generation[0..12]: [75, 75, 80, 78, 82, 81, 77, 82, 80, 83, 83, 83]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13426362Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13428033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"[2]R flush_numbers[0..12]: [119, 119, 131, 127, 135, 132, 126, 133, 129, 136, 136, 136]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13429604Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"[2]R generation[0..12]: [75, 75, 80, 78, 82, 81, 77, 82, 80, 83, 83, 83]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13431112Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13432745Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Max found gen is 84","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134340941Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Generation requested: 84 >= found:84","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134362908Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Next flush: 137","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134376499Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134389901Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"":"downstairs","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134404212Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134416754Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"client":"0","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134431055Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"client":"1","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134446965Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"client":"2","":"downstairs","session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134461736Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134475257Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"ac8e87af-5bc0-4f87-8220-5dbf0e136dfd is now active with session: a3c7fc68-1698-4fdb-8db0-cced6ed986b8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134490398Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.13450306Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541,"session_id":"a3c7fc68-1698-4fdb-8db0-cced6ed986b8"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134516061Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541} {"msg":"Activated sub_volume ac8e87af-5bc0-4f87-8220-5dbf0e136dfd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:04.134527224Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 162 len 28672 data: 4 4 3 4 3 4 4 02/30 Write block 319 len 20480 data: 4 5 4 4 5 03/30 Write block 476 len 32768 data: 6 8 10 9 10 10 9 8 04/30 Write block 124 len 4096 data: 7 05/30 Read block 61 len 8192 06/30 Read block 334 len 8192 07/30 Read block 217 len 20480 08/30 Write block 474 len 20480 data: 6 4 7 9 11 09/30 Read block 96 len 40960 10/30 Write block 571 len 36864 data: 6 6 4 5 8 9 11 11 10 11/30 Write block 283 len 12288 data: 3 5 6 12/30 Flush 13/30 Flush 14/30 Write block 343 len 16384 data: 5 5 6 7 15/30 Read block 543 len 12288 16/30 Flush 17/30 Write block 232 len 36864 data: 5 6 6 9 9 7 9 9 10 18/30 Flush 19/30 Write block 507 len 16384 data: 8 9 10 9 20/30 Write block 234 len 32768 data: 7 10 10 8 10 10 11 6 21/30 Read block 453 len 16384 22/30 Read block 352 len 32768 23/30 Write block 400 len 32768 data: 7 8 9 6 5 7 7 9 24/30 Read block 148 len 24576 25/30 Flush 26/30 Write block 451 len 12288 data: 7 6 7 27/30 Read block 97 len 28672 28/30 Read block 234 len 28672 29/30 Read block 145 len 24576 30/30 Read block 443 len 40960 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Read 2 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Read 5 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Write 9 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 4 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Read 4 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Read 6 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Read 7 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Read 10 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 0 27 Done 30 3 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:04.147049255Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1541} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 80 80 80 131 131 131 F F F 3 060-079 78 78 78 127 127 127 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 81 81 81 132 132 132 F F F 6 120-139 84 77 84 137 126 137 F F F <--- 7 140-159 82 82 82 133 133 133 F F F 8 160-179 84 80 84 137 129 137 F T F <--- 9 180-199 83 83 83 136 136 136 F F F 10 200-219 83 83 83 136 136 136 F F F 11 220-239 84 83 84 141 136 141 F F F <--- 12 240-259 84 82 84 141 134 141 F F F <--- 13 260-279 83 83 83 136 136 136 F F F 14 280-299 84 83 84 137 136 137 F F F <--- 15 300-319 84 78 84 137 127 137 F T F <--- 16 320-339 84 81 84 137 132 137 F T F <--- 17 340-359 84 81 84 139 132 139 F F F <--- 18 360-379 82 82 82 135 135 135 F F F 19 380-399 81 81 81 132 132 132 F F F 20 400-419 84 81 84 141 132 141 F F F <--- 21 420-439 82 82 82 133 133 133 F F F 22 440-459 79 79 79 128 128 128 T F T <--- 23 460-479 84 83 84 137 136 137 F T F <--- 24 480-499 84 82 84 137 135 137 F T F <--- 25 500-519 84 82 84 141 135 141 F F F <--- 26 520-539 81 81 81 132 132 132 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 84 83 84 137 136 137 F F F <--- 29 580-599 80 80 80 131 131 131 F F F Max gen: 84, Max flush: 141 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 80 80 80 131 131 131 F F F 3 060-079 78 78 78 127 127 127 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 81 81 81 132 132 132 F F F 6 120-139 84 84 84 137 137 137 F F F 7 140-159 82 82 82 133 133 133 F F F 8 160-179 84 84 84 137 137 137 F F F 9 180-199 83 83 83 136 136 136 F F F 10 200-219 83 83 83 136 136 136 F F F 11 220-239 84 84 84 141 141 141 F F F 12 240-259 84 84 84 141 141 141 F F F 13 260-279 83 83 83 136 136 136 F F F 14 280-299 84 84 84 137 137 137 F F F 15 300-319 84 84 84 137 137 137 F F F 16 320-339 84 84 84 137 137 137 F F F 17 340-359 84 84 84 139 139 139 F F F 18 360-379 82 82 82 135 135 135 F F F 19 380-399 81 81 81 132 132 132 F F F 20 400-419 84 84 84 141 141 141 F F F 21 420-439 82 82 82 133 133 133 F F F 22 440-459 85 85 85 142 142 142 F F F 23 460-479 84 84 84 137 137 137 F F F 24 480-499 84 84 84 137 137 137 F F F 25 500-519 84 84 84 141 141 141 F F F 26 520-539 81 81 81 132 132 132 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 84 84 84 137 137 137 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 85, Max flush: 142 Begin loop 43 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.466275084Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.477864281Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478177434Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.47821615Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478232309Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478250638Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549} {"msg":"Crucible 6681e9eb-a6b6-45b7-8ea2-7f62506bd97c has session id: 4a41bbab-7d50-4e09-8c37-805b1e10d120","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478593233Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Upstairs opts: Upstairs UUID: 6681e9eb-a6b6-45b7-8ea2-7f62506bd97c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478627251Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Crucible stats registered with UUID: 6681e9eb-a6b6-45b7-8ea2-7f62506bd97c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478643161Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:08.478664728Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478813414Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"io task","client":"2","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478848052Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"io task","client":"0","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478863323Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"io task","client":"1","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478967377Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"task":"crutest"} {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.478982788Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549} {"msg":"6681e9eb-a6b6-45b7-8ea2-7f62506bd97c active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.479000397Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.479013309Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"client":"0","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.47902671Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"client":"1","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.4790427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"client":"2","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.479056541Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"io task","client":"2","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.479070762Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"io task","client":"0","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.479084804Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"io task","client":"1","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"ds_connection connected from Ok(127.0.0.1:45219)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.479099754Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"io task","client":"2","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"ds_connection connected from Ok(127.0.0.1:54519)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.479114525Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"io task","client":"0","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"ds_connection connected from Ok(127.0.0.1:36894)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.479230932Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"io task","client":"1","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.47983786Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"client":"0","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.479876466Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"client":"2","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.479893715Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"client":"1","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480198993Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480242536Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"[0]R flush_numbers[0..12]: [119, 119, 131, 127, 135, 132, 137, 133, 137, 136, 136, 141]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480287278Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"[0]R generation[0..12]: [75, 75, 80, 78, 82, 81, 84, 82, 84, 83, 83, 84]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480303908Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480320427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"[1]R flush_numbers[0..12]: [119, 119, 131, 127, 135, 132, 137, 133, 137, 136, 136, 141]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480339475Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"[1]R generation[0..12]: [75, 75, 80, 78, 82, 81, 84, 82, 84, 83, 83, 84]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480355735Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480372255Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"[2]R flush_numbers[0..12]: [119, 119, 131, 127, 135, 132, 137, 133, 137, 136, 136, 141]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480387655Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"[2]R generation[0..12]: [75, 75, 80, 78, 82, 81, 84, 82, 84, 83, 83, 84]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480402566Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480418655Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Max found gen is 86","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480431997Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 86 >= found:86","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480451855Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Next flush: 143","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480465476Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480478998Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"":"downstairs","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480493259Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480505971Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"client":"0","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480520292Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"client":"1","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480534153Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"client":"2","":"downstairs","session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480550043Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480565853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"6681e9eb-a6b6-45b7-8ea2-7f62506bd97c is now active with session: 4a41bbab-7d50-4e09-8c37-805b1e10d120","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480581023Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480593645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549,"session_id":"4a41bbab-7d50-4e09-8c37-805b1e10d120"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.480606707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549} {"msg":"Activated sub_volume 6681e9eb-a6b6-45b7-8ea2-7f62506bd97c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:08.48061807Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 283 len 24576 data: 4 6 7 5 6 7 02/30 Flush 03/30 Write block 330 len 24576 data: 7 8 5 4 6 7 04/30 Read block 55 len 32768 05/30 Read block 169 len 4096 06/30 Write block 236 len 4096 data: 11 07/30 Read block 64 len 16384 08/30 Write block 244 len 28672 data: 9 10 11 13 15 13 15 09/30 Read block 290 len 40960 10/30 Read block 348 len 40960 11/30 Read block 282 len 28672 12/30 Read block 555 len 16384 13/30 Write block 45 len 28672 data: 5 3 4 4 5 5 5 14/30 Read block 102 len 16384 15/30 Read block 233 len 28672 16/30 Read block 54 len 24576 17/30 Read block 198 len 8192 18/30 Read block 232 len 4096 19/30 Read block 565 len 8192 20/30 Write block 195 len 28672 data: 10 6 6 4 6 6 7 21/30 Write block 292 len 16384 data: 13 10 8 7 22/30 Write block 147 len 16384 data: 5 5 4 5 23/30 Write block 264 len 28672 data: 8 8 8 7 7 8 9 24/30 Read block 42 len 20480 25/30 Read block 405 len 36864 26/30 Read block 86 len 4096 27/30 Read block 213 len 24576 28/30 Read block 288 len 36864 29/30 Read block 509 len 8192 30/30 Write block 2 len 12288 data: 5 7 6 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 6 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Read 1 Done Done Done false 1005 Acked Write 1 Done Done Done false 1006 Acked Read 4 Done Done Done false 1007 Acked Write 7 Done Done Done false 1008 Acked Read 10 Done Done Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Read 7 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Read 4 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Write 4 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 21 1 23 Done 27 7 27 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:08.49202785Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1549} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 75 75 75 119 119 119 T F T <--- 1 020-039 75 75 75 119 119 119 F F F 2 040-059 80 80 80 131 131 131 T F T <--- 3 060-079 78 78 78 127 127 127 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 81 81 81 132 132 132 F F F 6 120-139 84 84 84 137 137 137 F F F 7 140-159 82 82 82 133 133 133 T F T <--- 8 160-179 84 84 84 137 137 137 F F F 9 180-199 83 83 83 136 136 136 T F T <--- 10 200-219 83 83 83 136 136 136 T F T <--- 11 220-239 84 84 84 141 141 141 T T T 12 240-259 84 84 84 141 141 141 T T T 13 260-279 83 83 83 136 136 136 T F T <--- 14 280-299 86 86 86 143 143 143 T F T <--- 15 300-319 84 84 84 137 137 137 F F F 16 320-339 84 84 84 137 137 137 T T T 17 340-359 84 84 84 139 139 139 F F F 18 360-379 82 82 82 135 135 135 F F F 19 380-399 81 81 81 132 132 132 F F F 20 400-419 84 84 84 141 141 141 F F F 21 420-439 82 82 82 133 133 133 F F F 22 440-459 85 85 85 142 142 142 F F F 23 460-479 84 84 84 137 137 137 F F F 24 480-499 84 84 84 137 137 137 F F F 25 500-519 84 84 84 141 141 141 F F F 26 520-539 81 81 81 132 132 132 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 84 84 84 137 137 137 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 86, Max flush: 143 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 43 Verifying data now No change after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 87 87 87 144 144 144 F F F 1 020-039 75 75 75 119 119 119 F F F 2 040-059 87 87 87 144 144 144 F F F 3 060-079 78 78 78 127 127 127 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 81 81 81 132 132 132 F F F 6 120-139 84 84 84 137 137 137 F F F 7 140-159 87 87 87 144 144 144 F F F 8 160-179 84 84 84 137 137 137 F F F 9 180-199 87 87 87 144 144 144 F F F 10 200-219 87 87 87 144 144 144 F F F 11 220-239 87 87 87 144 144 144 F F F 12 240-259 87 87 87 144 144 144 F F F 13 260-279 87 87 87 144 144 144 F F F 14 280-299 87 87 87 144 144 144 F F F 15 300-319 84 84 84 137 137 137 F F F 16 320-339 87 87 87 144 144 144 F F F 17 340-359 84 84 84 139 139 139 F F F 18 360-379 82 82 82 135 135 135 F F F 19 380-399 81 81 81 132 132 132 F F F 20 400-419 84 84 84 141 141 141 F F F 21 420-439 82 82 82 133 133 133 F F F 22 440-459 85 85 85 142 142 142 F F F 23 460-479 84 84 84 137 137 137 F F F 24 480-499 84 84 84 137 137 137 F F F 25 500-519 84 84 84 141 141 141 F F F 26 520-539 81 81 81 132 132 132 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 84 84 84 137 137 137 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 87, Max flush: 144 Begin loop 44 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.812088014Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.823584009Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.823902889Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.823944933Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.823957405Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.823975074Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557} {"msg":"Crucible ec648ba8-5c8e-42e8-a1a2-2d0ef6f03ea8 has session id: 2678407e-773e-41e0-8d74-d6693f4caee1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.82429867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Upstairs opts: Upstairs UUID: ec648ba8-5c8e-42e8-a1a2-2d0ef6f03ea8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824325004Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Crucible stats registered with UUID: ec648ba8-5c8e-42e8-a1a2-2d0ef6f03ea8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824340254Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:12.824353356Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.82450645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"io task","client":"0","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824541458Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"io task","client":"2","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824557348Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"io task","client":"1","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824661053Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"task":"crutest"} {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824674834Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557} {"msg":"ec648ba8-5c8e-42e8-a1a2-2d0ef6f03ea8 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824694891Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824708003Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"client":"0","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824721884Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"client":"1","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824735766Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"client":"2","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824773882Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"io task","client":"2","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824799026Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"io task","client":"0","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824814356Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"io task","client":"1","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"ds_connection connected from Ok(127.0.0.1:49176)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824829277Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"io task","client":"1","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"ds_connection connected from Ok(127.0.0.1:53151)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824939737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"io task","client":"2","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"ds_connection connected from Ok(127.0.0.1:53596)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.824956927Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"io task","client":"0","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.825565824Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"client":"0","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.825587281Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"client":"1","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.82560381Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"client":"2","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.825918223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.825962964Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"[0]R flush_numbers[0..12]: [144, 119, 144, 127, 135, 132, 137, 144, 137, 144, 144, 144]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826009505Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"[0]R generation[0..12]: [87, 75, 87, 78, 82, 81, 84, 87, 84, 87, 87, 87]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826026225Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826042624Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"[1]R flush_numbers[0..12]: [144, 119, 144, 127, 135, 132, 137, 144, 137, 144, 144, 144]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826058574Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"[1]R generation[0..12]: [87, 75, 87, 78, 82, 81, 84, 87, 84, 87, 87, 87]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826073605Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826089645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"[2]R flush_numbers[0..12]: [144, 119, 144, 127, 135, 132, 137, 144, 137, 144, 144, 144]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826105265Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"[2]R generation[0..12]: [87, 75, 87, 78, 82, 81, 84, 87, 84, 87, 87, 87]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826119876Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826135666Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Max found gen is 88","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826148968Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826164868Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Next flush: 145","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826184255Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826197647Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"":"downstairs","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826211688Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.82622431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"client":"0","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826238691Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"client":"1","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826252363Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"client":"2","":"downstairs","session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826269232Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826282983Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"ec648ba8-5c8e-42e8-a1a2-2d0ef6f03ea8 is now active with session: 2678407e-773e-41e0-8d74-d6693f4caee1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826298074Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826310936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557,"session_id":"2678407e-773e-41e0-8d74-d6693f4caee1"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.826324697Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557} {"msg":"Activated sub_volume ec648ba8-5c8e-42e8-a1a2-2d0ef6f03ea8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:12.82633617Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 418 len 8192 data: 7 7 02/30 Flush 03/30 Flush 04/30 Write block 491 len 4096 data: 9 05/30 Flush 06/30 Write block 342 len 16384 data: 7 6 6 7 07/30 Read block 271 len 40960 08/30 Write block 427 len 12288 data: 5 3 4 09/30 Write block 421 len 8192 data: 7 5 10/30 Write block 161 len 40960 data: 5 5 5 4 5 4 5 5 4 4 11/30 Read block 247 len 4096 12/30 Write block 12 len 36864 data: 5 5 6 7 5 6 7 8 8 13/30 Read block 45 len 28672 14/30 Flush 15/30 Write block 438 len 28672 data: 8 8 9 9 11 12 12 16/30 Read block 52 len 28672 17/30 Read block 482 len 8192 18/30 Write block 285 len 16384 data: 8 6 7 8 19/30 Read block 480 len 36864 20/30 Write block 495 len 28672 data: 8 6 4 3 4 3 3 21/30 Write block 133 len 36864 data: 6 8 9 8 7 5 6 5 5 22/30 Write block 191 len 28672 data: 7 7 6 7 11 7 7 23/30 Write block 431 len 24576 data: 5 7 7 9 9 9 24/30 Read block 91 len 32768 25/30 Write block 498 len 16384 data: 4 5 4 4 26/30 Read block 454 len 28672 27/30 Write block 493 len 16384 data: 9 8 9 7 28/30 Read block 512 len 28672 29/30 Read block 202 len 32768 30/30 Read block 188 len 24576 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 4 Done Done Done false 1006 Acked Read 10 Done Done Done false 1007 Acked Write 3 Done Done Done false 1008 Acked Write 2 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 22 23 Done 25 24 3 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) Some(JobId(1013)) Some(JobId(1004)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:12.838513627Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1557} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 87 148 148 144 F F F <--- 1 020-039 88 88 75 148 148 119 F F F <--- 2 040-059 87 87 87 144 144 144 F F F 3 060-079 78 78 78 127 127 127 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 81 81 81 132 132 132 F F F 6 120-139 84 84 84 137 137 137 T T F <--- 7 140-159 87 87 87 144 144 144 T T F <--- 8 160-179 88 88 84 148 148 137 F F F <--- 9 180-199 87 87 87 144 144 144 T T F <--- 10 200-219 87 87 87 144 144 144 F F F 11 220-239 87 87 87 144 144 144 F F F 12 240-259 87 87 87 144 144 144 F F F 13 260-279 87 87 87 144 144 144 F F F 14 280-299 87 87 87 144 144 144 T T F <--- 15 300-319 84 84 84 137 137 137 F F F 16 320-339 87 87 87 144 144 144 F F F 17 340-359 88 88 84 148 148 139 F F T <--- 18 360-379 82 82 82 135 135 135 F F F 19 380-399 81 81 81 132 132 132 F F F 20 400-419 88 88 88 145 145 145 F F F 21 420-439 88 88 82 148 148 133 T T T <--- 22 440-459 85 85 85 142 142 142 T T F <--- 23 460-479 84 84 84 137 137 137 F F F 24 480-499 88 88 88 147 147 147 T T F <--- 25 500-519 84 84 84 141 141 141 T T F <--- 26 520-539 81 81 81 132 132 132 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 84 84 84 137 137 137 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 88, Max flush: 148 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 88 148 148 148 F F F 1 020-039 88 88 88 148 148 148 F F F 2 040-059 87 87 87 144 144 144 F F F 3 060-079 78 78 78 127 127 127 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 81 81 81 132 132 132 F F F 6 120-139 89 89 89 149 149 149 F F F 7 140-159 89 89 89 149 149 149 F F F 8 160-179 88 88 88 148 148 148 F F F 9 180-199 89 89 89 149 149 149 F F F 10 200-219 87 87 87 144 144 144 F F F 11 220-239 87 87 87 144 144 144 F F F 12 240-259 87 87 87 144 144 144 F F F 13 260-279 87 87 87 144 144 144 F F F 14 280-299 89 89 89 149 149 149 F F F 15 300-319 84 84 84 137 137 137 F F F 16 320-339 87 87 87 144 144 144 F F F 17 340-359 88 88 88 148 148 148 F F F 18 360-379 82 82 82 135 135 135 F F F 19 380-399 81 81 81 132 132 132 F F F 20 400-419 88 88 88 145 145 145 F F F 21 420-439 89 89 89 149 149 149 F F F 22 440-459 89 89 89 149 149 149 F F F 23 460-479 84 84 84 137 137 137 F F F 24 480-499 89 89 89 149 149 149 F F F 25 500-519 89 89 89 149 149 149 F F F 26 520-539 81 81 81 132 132 132 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 84 84 84 137 137 137 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 89, Max flush: 149 Begin loop 45 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.159299305Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.170913896Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.171230357Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.171271841Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.171286392Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.171304171Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565} {"msg":"Crucible 16dba464-b5e8-44e0-a96c-005b94349acf has session id: 6a5760e7-2644-4c0e-a055-cd2f22ef4aa9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.17163957Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Upstairs opts: Upstairs UUID: 16dba464-b5e8-44e0-a96c-005b94349acf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.171666283Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Crucible stats registered with UUID: 16dba464-b5e8-44e0-a96c-005b94349acf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.171681274Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:17.171694635Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.171861041Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"io task","client":"2","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.171897218Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"io task","client":"0","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.171912748Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"io task","client":"1","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172034971Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"task":"crutest"} {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172050292Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565} {"msg":"16dba464-b5e8-44e0-a96c-005b94349acf active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172067421Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172080323Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"client":"0","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172094344Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"client":"1","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172108255Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"client":"2","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172124315Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"io task","client":"2","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172138796Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"io task","client":"0","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172153377Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"io task","client":"1","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"ds_connection connected from Ok(127.0.0.1:60227)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172200617Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"io task","client":"0","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"ds_connection connected from Ok(127.0.0.1:39512)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172216797Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"io task","client":"2","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"ds_connection connected from Ok(127.0.0.1:37665)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.172316904Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"io task","client":"1","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.17297632Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"client":"2","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173004742Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"client":"0","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173021641Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"client":"1","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173360418Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173409378Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"[0]R flush_numbers[0..12]: [148, 148, 144, 127, 135, 132, 149, 149, 148, 149, 144, 144]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173460705Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"[0]R generation[0..12]: [88, 88, 87, 78, 82, 81, 89, 89, 88, 89, 87, 87]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173477475Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173494064Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"[1]R flush_numbers[0..12]: [148, 148, 144, 127, 135, 132, 149, 149, 148, 149, 144, 144]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173512643Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"[1]R generation[0..12]: [88, 88, 87, 78, 82, 81, 89, 89, 88, 89, 87, 87]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173528233Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173544413Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"[2]R flush_numbers[0..12]: [148, 148, 144, 127, 135, 132, 149, 149, 148, 149, 144, 144]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173560273Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"[2]R generation[0..12]: [88, 88, 87, 78, 82, 81, 89, 89, 88, 89, 87, 87]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173575633Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173592053Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173605644Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173625452Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Next flush: 150","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173638933Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173652305Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"":"downstairs","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173666476Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173679658Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"client":"0","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173693779Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"client":"1","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.17370763Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"client":"2","":"downstairs","session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:04:17.173721711Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173740669Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"16dba464-b5e8-44e0-a96c-005b94349acf is now active with session: 6a5760e7-2644-4c0e-a055-cd2f22ef4aa9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.17375551Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173768102Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565,"session_id":"6a5760e7-2644-4c0e-a055-cd2f22ef4aa9"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173781414Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565} {"msg":"Activated sub_volume 16dba464-b5e8-44e0-a96c-005b94349acf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:17.173792977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 233 len 36864 data: 7 8 11 12 9 11 11 12 7 02/30 Read block 275 len 12288 03/30 Read block 520 len 20480 04/30 Write block 68 len 28672 data: 5 6 7 7 7 7 11 05/30 Write block 526 len 4096 data: 7 06/30 Read block 368 len 12288 07/30 Read block 209 len 40960 08/30 Flush 09/30 Write block 520 len 40960 data: 6 5 6 6 5 7 8 9 9 8 10/30 Flush 11/30 Write block 420 len 40960 data: 7 8 6 7 6 6 5 6 4 5 12/30 Write block 140 len 24576 data: 6 6 5 5 6 4 13/30 Read block 154 len 8192 14/30 Read block 431 len 24576 15/30 Write block 238 len 40960 data: 12 12 13 8 7 8 10 11 12 14 16/30 Write block 339 len 20480 data: 7 6 6 8 7 17/30 Read block 315 len 36864 18/30 Read block 45 len 4096 19/30 Write block 523 len 36864 data: 7 6 8 9 10 10 9 8 9 20/30 Read block 509 len 24576 21/30 Write block 567 len 12288 data: 5 5 6 22/30 Write block 197 len 8192 data: 8 5 23/30 Flush 24/30 Write block 419 len 4096 data: 8 25/30 Write block 474 len 40960 data: 7 5 8 10 12 10 11 11 10 9 26/30 Read block 425 len 4096 27/30 Write block 326 len 16384 data: 8 8 8 6 28/30 Write block 477 len 20480 data: 11 13 11 12 12 29/30 Write block 393 len 8192 data: 8 8 30/30 Read block 469 len 40960 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Write 7 Done Done Done false 1004 Acked Write 1 Done Done Done false 1005 Acked Read 3 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 10 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Write 2 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Write 2 Done Sent Done false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 25 1 26 Done 30 5 29 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:17.187174497Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1565} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 88 148 148 148 F F F 1 020-039 88 88 88 148 148 148 F F F 2 040-059 87 87 87 144 144 144 F F F 3 060-079 90 78 90 150 127 150 F T F <--- 4 080-099 82 82 82 135 135 135 F F F 5 100-119 81 81 81 132 132 132 F F F 6 120-139 89 89 89 149 149 149 F F F 7 140-159 90 89 90 152 149 152 F F F <--- 8 160-179 88 88 88 148 148 148 F F F 9 180-199 90 89 90 152 149 152 F F F <--- 10 200-219 87 87 87 144 144 144 F F F 11 220-239 90 87 90 152 144 152 F T F <--- 12 240-259 90 87 90 152 144 152 F T F <--- 13 260-279 87 87 87 144 144 144 F F F 14 280-299 89 89 89 149 149 149 F F F 15 300-319 84 84 84 137 137 137 F F F 16 320-339 90 87 90 152 144 152 T F T <--- 17 340-359 90 88 90 152 148 152 F F F <--- 18 360-379 82 82 82 135 135 135 F F F 19 380-399 81 81 81 132 132 132 T F T <--- 20 400-419 88 88 88 145 145 145 T F T <--- 21 420-439 90 89 90 152 149 152 F F F <--- 22 440-459 89 89 89 149 149 149 F F F 23 460-479 84 84 84 137 137 137 T F T <--- 24 480-499 89 89 89 149 149 149 T F T <--- 25 500-519 89 89 89 149 149 149 F F F 26 520-539 90 81 90 152 132 152 F T F <--- 27 540-559 83 83 83 136 136 136 F F F 28 560-579 90 84 90 152 137 152 F F F <--- 29 580-599 80 80 80 131 131 131 F F F Max gen: 90, Max flush: 152 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 88 148 148 148 F F F 1 020-039 88 88 88 148 148 148 F F F 2 040-059 87 87 87 144 144 144 F F F 3 060-079 90 90 90 150 150 150 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 81 81 81 132 132 132 F F F 6 120-139 89 89 89 149 149 149 F F F 7 140-159 90 90 90 152 152 152 F F F 8 160-179 88 88 88 148 148 148 F F F 9 180-199 90 90 90 152 152 152 F F F 10 200-219 87 87 87 144 144 144 F F F 11 220-239 90 90 90 152 152 152 F F F 12 240-259 90 90 90 152 152 152 F F F 13 260-279 87 87 87 144 144 144 F F F 14 280-299 89 89 89 149 149 149 F F F 15 300-319 84 84 84 137 137 137 F F F 16 320-339 91 91 91 153 153 153 F F F 17 340-359 90 90 90 152 152 152 F F F 18 360-379 82 82 82 135 135 135 F F F 19 380-399 91 91 91 153 153 153 F F F 20 400-419 91 91 91 153 153 153 F F F 21 420-439 90 90 90 152 152 152 F F F 22 440-459 89 89 89 149 149 149 F F F 23 460-479 91 91 91 153 153 153 F F F 24 480-499 91 91 91 153 153 153 F F F 25 500-519 89 89 89 149 149 149 F F F 26 520-539 90 90 90 152 152 152 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 90 90 90 152 152 152 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 91, Max flush: 153 Begin loop 46 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.511069521Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.522644707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.52297495Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523017323Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523030305Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523048474Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573} {"msg":"Crucible 66c2dbf3-c985-4cf9-b574-b2cc8e2b26c9 has session id: 3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523394476Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Upstairs opts: Upstairs UUID: 66c2dbf3-c985-4cf9-b574-b2cc8e2b26c9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523428885Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Crucible stats registered with UUID: 66c2dbf3-c985-4cf9-b574-b2cc8e2b26c9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523444944Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:21.523461674Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523601536Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"io task","client":"2","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523625501Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"io task","client":"0","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523640292Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"io task","client":"1","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523751432Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"task":"crutest"} {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523767732Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573} {"msg":"66c2dbf3-c985-4cf9-b574-b2cc8e2b26c9 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.52378566Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523798282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"client":"0","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523812024Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"client":"1","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523825835Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"client":"2","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523839487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"io task","client":"2","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523853718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"io task","client":"0","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523867959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"io task","client":"1","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"ds_connection connected from Ok(127.0.0.1:45003)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523883039Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"io task","client":"2","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"ds_connection connected from Ok(127.0.0.1:60795)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.52389775Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"io task","client":"1","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"ds_connection connected from Ok(127.0.0.1:49130)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.523912531Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"io task","client":"0","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.524636696Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"client":"0","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.524679709Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"client":"1","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.524696818Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"client":"2","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525025861Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525061529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"[0]R flush_numbers[0..12]: [148, 148, 144, 150, 135, 132, 149, 152, 148, 152, 144, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525118703Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"[0]R generation[0..12]: [88, 88, 87, 90, 82, 81, 89, 90, 88, 90, 87, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525135642Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525152022Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"[1]R flush_numbers[0..12]: [148, 148, 144, 150, 135, 132, 149, 152, 148, 152, 144, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525167732Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"[1]R generation[0..12]: [88, 88, 87, 90, 82, 81, 89, 90, 88, 90, 87, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525183142Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525199282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"[2]R flush_numbers[0..12]: [148, 148, 144, 150, 135, 132, 149, 152, 148, 152, 144, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525214762Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"[2]R generation[0..12]: [88, 88, 87, 90, 82, 81, 89, 90, 88, 90, 87, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525229923Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525246133Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:04:21.525262043Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.52528183Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Next flush: 154","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525295132Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525308324Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"":"downstairs","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525322605Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525335396Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"client":"0","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525349688Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"client":"1","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525363519Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"client":"2","":"downstairs","session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:04:21.52537782Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525391921Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"66c2dbf3-c985-4cf9-b574-b2cc8e2b26c9 is now active with session: 3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525406872Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525419364Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573,"session_id":"3f47d24e-6310-4d7f-a98e-ee6b0a3ed9fb"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525432645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573} {"msg":"Activated sub_volume 66c2dbf3-c985-4cf9-b574-b2cc8e2b26c9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:21.525443628Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 139 len 32768 data: 7 7 7 6 6 7 5 6 02/30 Flush 03/30 Write block 199 len 32768 data: 7 7 8 7 11 9 11 9 04/30 Write block 244 len 28672 data: 11 12 13 15 16 14 16 05/30 Write block 245 len 4096 data: 13 06/30 Flush 07/30 Write block 370 len 16384 data: 6 5 4 4 08/30 Read block 39 len 28672 09/30 Read block 510 len 24576 10/30 Read block 93 len 8192 11/30 Read block 548 len 8192 12/30 Read block 180 len 28672 13/30 Read block 451 len 12288 14/30 Write block 498 len 16384 data: 5 6 5 5 15/30 Read block 430 len 32768 16/30 Write block 512 len 4096 data: 9 17/30 Write block 65 len 28672 data: 5 5 4 6 7 8 8 18/30 Read block 5 len 40960 19/30 Write block 533 len 16384 data: 8 8 6 5 20/30 Write block 100 len 36864 data: 3 5 6 7 6 6 7 6 6 21/30 Write block 64 len 36864 data: 5 6 6 5 7 8 9 9 8 22/30 Read block 422 len 36864 23/30 Read block 595 len 20480 24/30 Read block 372 len 12288 25/30 Read block 167 len 20480 26/30 Read block 176 len 8192 27/30 Write block 319 len 40960 data: 5 6 5 5 6 7 8 9 9 9 28/30 Read block 184 len 36864 29/30 Read block 144 len 8192 30/30 Read block 153 len 32768 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:24 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1006 Acked Write 4 Done Done Done false 1007 Acked Read 7 Done Done Done false 1008 Acked Read 6 Done Done Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Read 2 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Write 4 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Write 9 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Read 2 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Read 8 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 21 0 22 Done 23 3 24 50 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1005)) Some(JobId(1005)) Some(JobId(1005)) Downstairs last five completed: 1005 1004 1003 1002 1001 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:21.537748856Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1573} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 88 148 148 148 F F F 1 020-039 88 88 88 148 148 148 F F F 2 040-059 87 87 87 144 144 144 F F F 3 060-079 90 90 90 150 150 150 T F T <--- 4 080-099 82 82 82 135 135 135 F F F 5 100-119 81 81 81 132 132 132 T F T <--- 6 120-139 92 92 92 154 154 154 F F F 7 140-159 92 92 92 154 154 154 F F F 8 160-179 88 88 88 148 148 148 F F F 9 180-199 92 92 92 155 155 155 F F F 10 200-219 92 92 92 155 155 155 F F F 11 220-239 90 90 90 152 152 152 F F F 12 240-259 92 92 92 155 155 155 F F F 13 260-279 87 87 87 144 144 144 F F F 14 280-299 89 89 89 149 149 149 F F F 15 300-319 84 84 84 137 137 137 T F T <--- 16 320-339 91 91 91 153 153 153 T F T <--- 17 340-359 90 90 90 152 152 152 F F F 18 360-379 82 82 82 135 135 135 T T T 19 380-399 91 91 91 153 153 153 F F F 20 400-419 91 91 91 153 153 153 F F F 21 420-439 90 90 90 152 152 152 F F F 22 440-459 89 89 89 149 149 149 F F F 23 460-479 91 91 91 153 153 153 F F F 24 480-499 91 91 91 153 153 153 T F T <--- 25 500-519 89 89 89 149 149 149 T F T <--- 26 520-539 90 90 90 152 152 152 T F T <--- 27 540-559 83 83 83 136 136 136 F F F 28 560-579 90 90 90 152 152 152 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 92, Max flush: 155 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 88 148 148 148 F F F 1 020-039 88 88 88 148 148 148 F F F 2 040-059 87 87 87 144 144 144 F F F 3 060-079 93 93 93 156 156 156 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 93 93 93 156 156 156 F F F 6 120-139 92 92 92 154 154 154 F F F 7 140-159 92 92 92 154 154 154 F F F 8 160-179 88 88 88 148 148 148 F F F 9 180-199 92 92 92 155 155 155 F F F 10 200-219 92 92 92 155 155 155 F F F 11 220-239 90 90 90 152 152 152 F F F 12 240-259 92 92 92 155 155 155 F F F 13 260-279 87 87 87 144 144 144 F F F 14 280-299 89 89 89 149 149 149 F F F 15 300-319 93 93 93 156 156 156 F F F 16 320-339 93 93 93 156 156 156 F F F 17 340-359 90 90 90 152 152 152 F F F 18 360-379 93 93 93 156 156 156 F F F 19 380-399 91 91 91 153 153 153 F F F 20 400-419 91 91 91 153 153 153 F F F 21 420-439 90 90 90 152 152 152 F F F 22 440-459 89 89 89 149 149 149 F F F 23 460-479 91 91 91 153 153 153 F F F 24 480-499 93 93 93 156 156 156 F F F 25 500-519 93 93 93 156 156 156 F F F 26 520-539 93 93 93 156 156 156 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 90 90 90 152 152 152 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 93, Max flush: 156 Begin loop 47 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.849150659Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.86079674Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.86113122Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861172824Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861189114Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861207192Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581} {"msg":"Crucible a5a4a917-a8ae-44b2-a993-a61979b7f467 has session id: 46d10258-9d8c-43b9-bcb2-ec365d716dc9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861528511Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Upstairs opts: Upstairs UUID: a5a4a917-a8ae-44b2-a993-a61979b7f467, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861555793Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Crucible stats registered with UUID: a5a4a917-a8ae-44b2-a993-a61979b7f467","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861570724Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:25.861583936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861755378Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"io task","client":"2","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861788637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"io task","client":"0","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861803967Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"io task","client":"1","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861907952Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"task":"crutest"} {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861928639Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581} {"msg":"a5a4a917-a8ae-44b2-a993-a61979b7f467 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.861978478Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862003922Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"client":"0","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862019502Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"client":"1","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862036871Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"client":"2","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862096034Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"io task","client":"2","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862122877Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"io task","client":"0","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862138767Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"io task","client":"1","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"ds_connection connected from Ok(127.0.0.1:36058)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862154377Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"io task","client":"0","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"ds_connection connected from Ok(127.0.0.1:55251)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862219696Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"io task","client":"2","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"ds_connection connected from Ok(127.0.0.1:36119)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862237475Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"io task","client":"1","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862934357Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"client":"1","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862971134Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"client":"0","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.862988543Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"client":"2","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863332517Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863363437Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"[0]R flush_numbers[0..12]: [148, 148, 144, 156, 135, 156, 154, 154, 148, 155, 155, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863448734Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"[0]R generation[0..12]: [88, 88, 87, 93, 82, 93, 92, 92, 88, 92, 92, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863466223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863483002Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"[1]R flush_numbers[0..12]: [148, 148, 144, 156, 135, 156, 154, 154, 148, 155, 155, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863501541Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"[1]R generation[0..12]: [88, 88, 87, 93, 82, 93, 92, 92, 88, 92, 92, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863516931Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863533371Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"[2]R flush_numbers[0..12]: [148, 148, 144, 156, 135, 156, 154, 154, 148, 155, 155, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863549301Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"[2]R generation[0..12]: [88, 88, 87, 93, 82, 93, 92, 92, 88, 92, 92, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863564761Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863581191Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Max found gen is 94","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863594742Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairsWait for a query_work_queue command to finish before sending IO ","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Generation requested: 94 >= found:94","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863615609Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Next flush: 157","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863628781Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863648029Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"":"downstairs","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.86366248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863675232Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"client":"0","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863689513Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"client":"1","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863703504Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"client":"2","":"downstairs","session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863720793Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863735913Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"a5a4a917-a8ae-44b2-a993-a61979b7f467 is now active with session: 46d10258-9d8c-43b9-bcb2-ec365d716dc9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863750974Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863763906Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581,"session_id":"46d10258-9d8c-43b9-bcb2-ec365d716dc9"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.863777577Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581} {"msg":"Activated sub_volume a5a4a917-a8ae-44b2-a993-a61979b7f467","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:25.86378929Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 27 len 24576 data: 4 7 7 8 8 7 02/30 Read block 517 len 4096 03/30 Write block 347 len 40960 data: 7 8 9 10 9 10 8 7 6 8 04/30 Read block 305 len 20480 05/30 Flush 06/30 Flush 07/30 Write block 355 len 24576 data: 7 9 8 7 6 9 08/30 Read block 292 len 8192 09/30 Flush 10/30 Write block 449 len 4096 data: 8 11/30 Read block 560 len 32768 12/30 Flush 13/30 Flush 14/30 Write block 396 len 24576 data: 8 8 7 8 8 9 15/30 Read block 491 len 16384 16/30 Write block 184 len 32768 data: 8 9 8 9 9 7 7 8 17/30 Write block 268 len 36864 data: 8 9 10 7 7 6 8 8 7 18/30 Read block 140 len 40960 19/30 Write block 429 len 32768 data: 6 4 6 8 8 10 10 10 20/30 Read block 159 len 8192 21/30 Write block 36 len 40960 data: 9 7 5 6 5 5 5 4 4 6 22/30 Read block 276 len 8192 23/30 Read block 179 len 20480 24/30 Flush 25/30 Write block 208 len 16384 data: 11 11 9 8 26/30 Read block 348 len 36864 27/30 Write block 166 len 36864 data: 5 6 6 5 5 3 4 4 4 28/30 Read block 294 len 20480 29/30 Read block 172 len 36864 30/30 Write block 178 len 16384 data: 8 7 5 5 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Sent false 1001 Acked Read 1 Done Done Sent false 1002 Acked Write 10 Done Done Sent false 1003 Acked Read 5 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Read 2 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Read 5 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:25.877759171Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1581} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 88 148 148 148 F F F 1 020-039 94 94 88 162 162 148 F F F <--- 2 040-059 94 94 87 162 162 144 F F F <--- 3 060-079 93 93 93 156 156 156 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 93 93 93 156 156 156 F F F 6 120-139 92 92 92 154 154 154 F F F 7 140-159 92 92 92 154 154 154 F F F 8 160-179 88 88 88 148 148 148 T T F <--- 9 180-199 94 94 92 162 162 155 T T F <--- 10 200-219 92 92 92 155 155 155 T T F <--- 11 220-239 90 90 90 152 152 152 F F F 12 240-259 92 92 92 155 155 155 F F F 13 260-279 94 94 87 162 162 144 F F F <--- 14 280-299 89 89 89 149 149 149 F F F 15 300-319 93 93 93 156 156 156 F F F 16 320-339 93 93 93 156 156 156 F F F 17 340-359 94 94 90 159 159 152 F F F <--- 18 360-379 94 94 93 159 159 156 F F F <--- 19 380-399 94 94 91 162 162 153 F F F <--- 20 400-419 94 94 91 162 162 153 F F F <--- 21 420-439 94 94 90 162 162 152 F F F <--- 22 440-459 94 94 89 160 160 149 F F F <--- 23 460-479 91 91 91 153 153 153 F F F 24 480-499 93 93 93 156 156 156 F F F 25 500-519 93 93 93 156 156 156 F F F 26 520-539 93 93 93 156 156 156 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 90 90 90 152 152 152 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 94, Max flush: 162 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 47 Verifying data now No change after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 88 148 148 148 F F F 1 020-039 94 94 94 162 162 162 F F F 2 040-059 94 94 94 162 162 162 F F F 3 060-079 93 93 93 156 156 156 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 93 93 93 156 156 156 F F F 6 120-139 92 92 92 154 154 154 F F F 7 140-159 92 92 92 154 154 154 F F F 8 160-179 95 95 95 163 163 163 F F F 9 180-199 95 95 95 163 163 163 F F F 10 200-219 95 95 95 163 163 163 F F F 11 220-239 90 90 90 152 152 152 F F F 12 240-259 92 92 92 155 155 155 F F F 13 260-279 94 94 94 162 162 162 F F F 14 280-299 89 89 89 149 149 149 F F F 15 300-319 93 93 93 156 156 156 F F F 16 320-339 93 93 93 156 156 156 F F F 17 340-359 94 94 94 159 159 159 F F F 18 360-379 94 94 94 159 159 159 F F F 19 380-399 94 94 94 162 162 162 F F F 20 400-419 94 94 94 162 162 162 F F F 21 420-439 94 94 94 162 162 162 F F F 22 440-459 94 94 94 160 160 160 F F F 23 460-479 91 91 91 153 153 153 F F F 24 480-499 93 93 93 156 156 156 F F F 25 500-519 93 93 93 156 156 156 F F F 26 520-539 93 93 93 156 156 156 F F F 27 540-559 83 83 83 136 136 136 F F F 28 560-579 90 90 90 152 152 152 F F F 29 580-599 80 80 80 131 131 131 F F F Max gen: 95, Max flush: 163 Begin loop 48 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.197644364Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.209224247Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.209540188Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.209577705Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.209589957Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.209607916Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589} {"msg":"Crucible 6a84971e-af23-4528-84f6-c2e6d13a10e2 has session id: 7fc071af-aee1-4320-96b0-51ec2966b2ab","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.209939837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Upstairs opts: Upstairs UUID: 6a84971e-af23-4528-84f6-c2e6d13a10e2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.209974945Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Crucible stats registered with UUID: 6a84971e-af23-4528-84f6-c2e6d13a10e2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.209990495Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:30.210015469Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210171391Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"io task","client":"1","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210196026Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"io task","client":"0","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210210777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"io task","client":"2","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210335288Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"task":"crutest"} {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210350689Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589} {"msg":"6a84971e-af23-4528-84f6-c2e6d13a10e2 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210367648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.21038041Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"client":"0","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210394831Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"client":"1","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210408692Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"client":"2","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210422304Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"io task","client":"2","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210436695Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"io task","client":"0","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210451036Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"io task","client":"1","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:43251)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.21049178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"io task","client":"0","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:57070)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.21050775Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"io task","client":"2","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"ds_connection connected from Ok(127.0.0.1:40943)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.210522821Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"io task","client":"1","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211162648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"client":"2","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211188023Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"client":"0","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211228067Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"client":"1","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211530307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211544329Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"[0]R flush_numbers[0..12]: [148, 162, 162, 156, 135, 156, 154, 154, 163, 163, 163, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211642567Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"[0]R generation[0..12]: [88, 94, 94, 93, 82, 93, 92, 92, 95, 95, 95, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211659047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211675296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"[1]R flush_numbers[0..12]: [148, 162, 162, 156, 135, 156, 154, 154, 163, 163, 163, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211691376Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"[1]R generation[0..12]: [88, 94, 94, 93, 82, 93, 92, 92, 95, 95, 95, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211706557Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211723126Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"[2]R flush_numbers[0..12]: [148, 162, 162, 156, 135, 156, 154, 154, 163, 163, 163, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211740196Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"[2]R generation[0..12]: [88, 94, 94, 93, 82, 93, 92, 92, 95, 95, 95, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211755636Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211771666Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211787156Z"Wait for a query_work_queue command to finish before sending IO ,"hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211806834Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Next flush: 164","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211820295Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211833707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"":"downstairs","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211848248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211861339Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"client":"0","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.21187559Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"client":"1","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211889342Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"client":"2","":"downstairs","session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211906201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211920152Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"6a84971e-af23-4528-84f6-c2e6d13a10e2 is now active with session: 7fc071af-aee1-4320-96b0-51ec2966b2ab","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211935173Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211948025Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589,"session_id":"7fc071af-aee1-4320-96b0-51ec2966b2ab"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211961696Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589} {"msg":"Activated sub_volume 6a84971e-af23-4528-84f6-c2e6d13a10e2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:30.211973579Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 169 len 8192 data: 6 6 02/30 Read block 503 len 36864 03/30 Flush 04/30 Write block 411 len 24576 data: 5 5 6 7 5 6 05/30 Read block 315 len 28672 06/30 Read block 394 len 20480 07/30 Write block 516 len 28672 data: 7 7 7 8 7 6 7 08/30 Write block 538 len 8192 data: 3 4 09/30 Write block 146 len 12288 data: 7 6 6 10/30 Read block 591 len 24576 11/30 Write block 514 len 32768 data: 7 10 8 8 8 9 8 7 12/30 Write block 545 len 28672 data: 4 5 7 8 7 8 7 13/30 Read block 212 len 28672 14/30 Read block 515 len 24576 15/30 Flush 16/30 Write block 583 len 28672 data: 12 11 8 7 6 6 8 17/30 Read block 586 len 36864 18/30 Read block 262 len 4096 19/30 Write block 157 len 16384 data: 7 8 7 5 20/30 Write block 280 len 24576 data: 5 4 5 5 7 9 21/30 Write block 583 len 20480 data: 13 12 9 8 7 22/30 Flush 23/30 Write block 163 len 4096 data: 6 24/30 Write block 101 len 36864 data: 6 7 8 7 7 8 7 7 8 25/30 Flush 26/30 Write block 59 len 32768 data: 7 9 8 6 6 6 7 7 27/30 Write block 174 len 28672 data: 5 4 7 8 9 8 6 28/30 Write block 58 len 20480 data: 5 8 10 9 7 29/30 Write block 453 len 32768 data: 8 7 10 9 8 8 9 7 30/30 Read block 580 len 4096 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Sent Done false 1001 Acked Read 9 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 6 Done Sent Done false 1004 Acked Read 7 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 6 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Write 4 Done Sent Done false 1019 Acked Write 6 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Write 8 Done Sent Done false 1029 Acked Read 1 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:30.223977375Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1589} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 88 148 148 148 F F F 1 020-039 94 94 94 162 162 162 F F F 2 040-059 94 94 94 162 162 162 T F T <--- 3 060-079 93 93 93 156 156 156 T F T <--- 4 080-099 82 82 82 135 135 135 F F F 5 100-119 96 93 96 167 156 167 F F F <--- 6 120-139 92 92 92 154 154 154 F F F 7 140-159 96 92 96 166 154 166 F F F <--- 8 160-179 96 95 96 167 163 167 T F T <--- 9 180-199 95 95 95 163 163 163 T F T <--- 10 200-219 95 95 95 163 163 163 F F F 11 220-239 90 90 90 152 152 152 F F F 12 240-259 92 92 92 155 155 155 F F F 13 260-279 94 94 94 162 162 162 F F F 14 280-299 96 89 96 166 149 166 F F F <--- 15 300-319 93 93 93 156 156 156 F F F 16 320-339 93 93 93 156 156 156 F F F 17 340-359 94 94 94 159 159 159 F F F 18 360-379 94 94 94 159 159 159 F F F 19 380-399 94 94 94 162 162 162 F F F 20 400-419 96 94 96 165 162 165 F F F <--- 21 420-439 94 94 94 162 162 162 F F F 22 440-459 94 94 94 160 160 160 T F T <--- 23 460-479 91 91 91 153 153 153 T F T <--- 24 480-499 93 93 93 156 156 156 F F F 25 500-519 96 93 96 165 156 165 F F F <--- 26 520-539 96 93 96 165 156 165 F F F <--- 27 540-559 96 83 96 165 136 165 F F F <--- 28 560-579 90 90 90 152 152 152 F F F 29 580-599 96 80 96 166 131 166 F F F <--- Max gen: 96, Max flush: 167 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 88 88 148 148 148 F F F 1 020-039 94 94 94 162 162 162 F F F 2 040-059 97 97 97 168 168 168 F F F 3 060-079 97 97 97 168 168 168 F F F 4 080-099 82 82 82 135 135 135 F F F 5 100-119 96 96 96 167 167 167 F F F 6 120-139 92 92 92 154 154 154 F F F 7 140-159 96 96 96 166 166 166 F F F 8 160-179 97 97 97 168 168 168 F F F 9 180-199 97 97 97 168 168 168 F F F 10 200-219 95 95 95 163 163 163 F F F 11 220-239 90 90 90 152 152 152 F F F 12 240-259 92 92 92 155 155 155 F F F 13 260-279 94 94 94 162 162 162 F F F 14 280-299 96 96 96 166 166 166 F F F 15 300-319 93 93 93 156 156 156 F F F 16 320-339 93 93 93 156 156 156 F F F 17 340-359 94 94 94 159 159 159 F F F 18 360-379 94 94 94 159 159 159 F F F 19 380-399 94 94 94 162 162 162 F F F 20 400-419 96 96 96 165 165 165 F F F 21 420-439 94 94 94 162 162 162 F F F 22 440-459 97 97 97 168 168 168 F F F 23 460-479 97 97 97 168 168 168 F F F 24 480-499 93 93 93 156 156 156 F F F 25 500-519 96 96 96 165 165 165 F F F 26 520-539 96 96 96 165 165 165 F F F 27 540-559 96 96 96 165 165 165 F F F 28 560-579 90 90 90 152 152 152 F F F 29 580-599 96 96 96 166 166 166 F F F Max gen: 97, Max flush: 168 Begin loop 49 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.551419451Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:34.562076734Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.562122525Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.572482145Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.572787024Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.572826359Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.572838182Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.57285578Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597} {"msg":"Crucible 98956782-03a6-41df-88a1-b6f1e6ace5cd has session id: 3f9edf64-4de1-458a-b0d7-35ca0edbbdc4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573220201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Upstairs opts: Upstairs UUID: 98956782-03a6-41df-88a1-b6f1e6ace5cd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573255139Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Crucible stats registered with UUID: 98956782-03a6-41df-88a1-b6f1e6ace5cd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.57327088Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:34.573284371Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573424843Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"io task","client":"0","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573449188Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"io task","client":"1","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573463918Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"io task","client":"2","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573584992Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"task":"crutest"} {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573599223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597} {"msg":"98956782-03a6-41df-88a1-b6f1e6ace5cd active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573618101Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573630983Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"client":"0","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573645104Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"client":"1","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573659266Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"client":"2","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573672817Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"io task","client":"2","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573686988Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"io task","client":"0","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573701309Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"io task","client":"1","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"ds_connection connected from Ok(127.0.0.1:39675)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573799148Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"io task","client":"2","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"ds_connection connected from Ok(127.0.0.1:46863)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573952141Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"io task","client":"1","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"ds_connection connected from Ok(127.0.0.1:50576)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.573978535Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"io task","client":"0","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.574490503Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"client":"2","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.574510551Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"client":"1","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.57452674Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"client":"0","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.574851556Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.574895649Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"[0]R flush_numbers[0..12]: [148, 162, 168, 168, 135, 167, 154, 166, 168, 168, 163, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.574962187Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"[0]R generation[0..12]: [88, 94, 97, 97, 82, 96, 92, 96, 97, 97, 95, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.574981864Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.574998704Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"[1]R flush_numbers[0..12]: [148, 162, 168, 168, 135, 167, 154, 166, 168, 168, 163, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575014714Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"[1]R generation[0..12]: [88, 94, 97, 97, 82, 96, 92, 96, 97, 97, 95, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575029934Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575046264Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"[2]R flush_numbers[0..12]: [148, 162, 168, 168, 135, 167, 154, 166, 168, 168, 163, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575062084Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"[2]R generation[0..12]: [88, 94, 97, 97, 82, 96, 92, 96, 97, 97, 95, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575077424Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575093874Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575107605Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575121637Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Next flush: 169","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575140894Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575154266Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"":"downstairs","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575168237Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575180979Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"client":"0","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575196649Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"client":"1","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575210571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"client":"2","":"downstairs","session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575224462Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575239632Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"98956782-03a6-41df-88a1-b6f1e6ace5cd is now active with session: 3f9edf64-4de1-458a-b0d7-35ca0edbbdc4","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575254603Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575268055Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597,"session_id":"3f9edf64-4de1-458a-b0d7-35ca0edbbdc4"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575280996Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597} {"msg":"Activated sub_volume 98956782-03a6-41df-88a1-b6f1e6ace5cd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:34.575292889Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 21 len 12288 data: 7 7 4 03/30 Read block 35 len 36864 04/30 Read block 407 len 8192 05/30 Read block 55 len 4096 06/30 Write block 134 len 20480 data: 9 10 9 8 6 07/30 Write block 367 len 40960 data: 10 9 8 7 6 5 5 4 5 5 08/30 Read block 473 len 28672 09/30 Write block 538 len 8192 data: 4 5 10/30 Write block 524 len 40960 data: 7 9 10 11 11 10 9 10 9 9 11/30 Flush 12/30 Write block 370 len 16384 data: 8 7 6 6 13/30 Flush 14/30 Write block 143 len 8192 data: 7 8 15/30 Read block 409 len 36864 16/30 Flush 17/30 Flush 18/30 Write block 311 len 40960 data: 13 15 13 11 12 9 7 5 6 7 19/30 Read block 518 len 20480 20/30 Write block 33 len 24576 data: 9 8 9 10 8 6 21/30 Flush 22/30 Write block 19 len 20480 data: 9 9 8 8 5 23/30 Read block 427 len 8192 24/30 Flush 25/30 Write block 162 len 28672 data: 6 7 5 6 6 7 7 26/30 Read block 593 len 8192 27/30 Write block 97 len 40960 data: 4 2 4 4 7 8 9 8 8 9 28/30 Write block 572 len 4096 data: 7 29/30 Read block 299 len 36864 30/30 Read block 114 len 8192 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Sent Done Done false 1002 Acked Read 9 Sent Done Done false 1003 Acked Read 2 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Write 10 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 2 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 2 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Read 2 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:34.58818118Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1597} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 88 98 98 148 175 175 F F F <--- 1 020-039 94 98 98 162 175 175 F F F <--- 2 040-059 97 97 97 168 168 168 F F F 3 060-079 97 97 97 168 168 168 F F F 4 080-099 82 82 82 135 135 135 F T T <--- 5 100-119 96 96 96 167 167 167 F T T <--- 6 120-139 92 98 98 154 170 170 F F F <--- 7 140-159 96 98 98 166 172 172 F F F <--- 8 160-179 97 97 97 168 168 168 F T T <--- 9 180-199 97 97 97 168 168 168 F F F 10 200-219 95 95 95 163 163 163 F F F 11 220-239 90 90 90 152 152 152 F F F 12 240-259 92 92 92 155 155 155 F F F 13 260-279 94 94 94 162 162 162 F F F 14 280-299 96 96 96 166 166 166 F F F 15 300-319 93 98 98 156 174 174 F F F <--- 16 320-339 93 98 98 156 174 174 F F F <--- 17 340-359 94 94 94 159 159 159 F F F 18 360-379 94 98 98 159 171 171 F F F <--- 19 380-399 94 94 94 162 162 162 F F F 20 400-419 96 96 96 165 165 165 F F F 21 420-439 94 94 94 162 162 162 F F F 22 440-459 97 97 97 168 168 168 F F F 23 460-479 97 97 97 168 168 168 F F F 24 480-499 93 93 93 156 156 156 F F F 25 500-519 96 96 96 165 165 165 F F F 26 520-539 96 98 98 165 170 170 F F F <--- 27 540-559 96 96 96 165 165 165 F F F 28 560-579 90 90 90 152 152 152 F T T <--- 29 580-599 96 96 96 166 166 166 F F F Max gen: 98, Max flush: 175 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 98 98 175 175 175 F F F 1 020-039 98 98 98 175 175 175 F F F 2 040-059 97 97 97 168 168 168 F F F 3 060-079 97 97 97 168 168 168 F F F 4 080-099 99 99 99 176 176 176 F F F 5 100-119 99 99 99 176 176 176 F F F 6 120-139 98 98 98 170 170 170 F F F 7 140-159 98 98 98 172 172 172 F F F 8 160-179 99 99 99 176 176 176 F F F 9 180-199 97 97 97 168 168 168 F F F 10 200-219 95 95 95 163 163 163 F F F 11 220-239 90 90 90 152 152 152 F F F 12 240-259 92 92 92 155 155 155 F F F 13 260-279 94 94 94 162 162 162 F F F 14 280-299 96 96 96 166 166 166 F F F 15 300-319 98 98 98 174 174 174 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 94 94 94 159 159 159 F F F 18 360-379 98 98 98 171 171 171 F F F 19 380-399 94 94 94 162 162 162 F F F 20 400-419 96 96 96 165 165 165 F F F 21 420-439 94 94 94 162 162 162 F F F 22 440-459 97 97 97 168 168 168 F F F 23 460-479 97 97 97 168 168 168 F F F 24 480-499 93 93 93 156 156 156 F F F 25 500-519 96 96 96 165 165 165 F F F 26 520-539 98 98 98 170 170 170 F F F 27 540-559 96 96 96 165 165 165 F F F 28 560-579 99 99 99 176 176 176 F F F 29 580-599 96 96 96 166 166 166 F F F Max gen: 99, Max flush: 176 Begin loop 50 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.919244807Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.931075333Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.931391544Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.93143074Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.931442922Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.931460971Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605} {"msg":"Crucible 238e1d4b-f0ef-4a1d-bc4d-b6022590e12c has session id: 6471d557-6799-4ace-90a1-f45d6a89c4a8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.931815997Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Upstairs opts: Upstairs UUID: 238e1d4b-f0ef-4a1d-bc4d-b6022590e12c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.931850096Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Crucible stats registered with UUID: 238e1d4b-f0ef-4a1d-bc4d-b6022590e12c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.931868994Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:38.931882516Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932029563Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"io task","client":"2","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932053238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"io task","client":"0","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932067669Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"io task","client":"1","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932164538Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"task":"crutest"} {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932179919Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605} {"msg":"238e1d4b-f0ef-4a1d-bc4d-b6022590e12c active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.93221031Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932224241Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"client":"0","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932238162Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"client":"1","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932252163Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"client":"2","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932265805Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"io task","client":"2","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932280286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"io task","client":"0","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932294717Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"io task","client":"1","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"ds_connection connected from Ok(127.0.0.1:64227)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932309877Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"io task","client":"2","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"ds_connection connected from Ok(127.0.0.1:44004)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932333502Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"io task","client":"0","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"ds_connection connected from Ok(127.0.0.1:65330)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.932348573Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"io task","client":"1","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.93302069Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"client":"1","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933048503Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"client":"0","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933168987Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"client":"2","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933388689Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933427894Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"[0]R flush_numbers[0..12]: [175, 175, 168, 168, 176, 176, 170, 172, 176, 168, 163, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933516749Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"[0]R generation[0..12]: [98, 98, 97, 97, 99, 99, 98, 98, 99, 97, 95, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933533518Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933549948Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"[1]R flush_numbers[0..12]: [175, 175, 168, 168, 176, 176, 170, 172, 176, 168, 163, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933565728Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"[1]R generation[0..12]: [98, 98, 97, 97, 99, 99, 98, 98, 99, 97, 95, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933581028Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933597458Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"[2]R flush_numbers[0..12]: [175, 175, 168, 168, 176, 176, 170, 172, 176, 168, 163, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933613068Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"[2]R generation[0..12]: [98, 98, 97, 97, 99, 99, 98, 98, 99, 97, 95, 90]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933630817Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933647526Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933661228Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933675219Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid"Wait for a query_work_queue command to finish before sending IO :1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Next flush: 177","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933693807Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933707279Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"":"downstairs","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.93372135Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933734032Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"client":"0","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933748033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"client":"1","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933762005Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"client":"2","":"downstairs","session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933776455Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:04:38.933789797Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"238e1d4b-f0ef-4a1d-bc4d-b6022590e12c is now active with session: 6471d557-6799-4ace-90a1-f45d6a89c4a8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933805987Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933819189Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605,"session_id":"6471d557-6799-4ace-90a1-f45d6a89c4a8"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.93383263Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605} {"msg":"Activated sub_volume 238e1d4b-f0ef-4a1d-bc4d-b6022590e12c","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:38.933844403Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 62 len 40960 data: 8 7 7 8 8 6 8 9 10 10 02/30 Read block 193 len 4096 03/30 Write block 511 len 4096 data: 8 04/30 Read block 198 len 32768 05/30 Write block 347 len 16384 data: 8 9 10 11 06/30 Read block 125 len 12288 07/30 Flush 08/30 Write block 24 len 40960 data: 4 3 2 5 8 8 9 9 8 10 09/30 Read block 347 len 12288 10/30 Read block 45 len 12288 11/30 Read block 577 len 40960 12/30 Write block 175 len 16384 data: 5 8 9 10 13/30 Read block 45 len 40960 14/30 Read block 408 len 36864 15/30 Write block 174 len 32768 data: 6 6 9 10 11 9 7 6 16/30 Write block 207 len 8192 data: 9 12 17/30 Write block 217 len 20480 data: 12 7 7 7 6 18/30 Read block 556 len 8192 19/30 Flush 20/30 Write block 102 len 8192 data: 9 10 21/30 Write block 365 len 20480 data: 6 9 11 10 9 22/30 Write block 103 len 8192 data: 11 9 23/30 Read block 502 len 28672 24/30 Read block 467 len 20480 25/30 Write block 16 len 16384 data: 6 7 8 10 26/30 Read block 270 len 20480 27/30 Read block 140 len 8192 28/30 Write block 249 len 4096 data: 15 29/30 Read block 449 len 20480 30/30 Write block 275 len 40960 data: 9 8 5 5 6 6 5 6 6 8 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Write 4 Done Done Sent false 1005 Acked Read 3 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Read 10 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Read 9 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 5 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Write 1 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 26 28 Done 29 29 4 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:38.945758286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1605} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 98 98 175 175 175 T T F <--- 1 020-039 100 100 98 178 178 175 F F F <--- 2 040-059 97 97 97 168 168 168 F F F 3 060-079 100 100 97 177 177 168 F F T <--- 4 080-099 99 99 99 176 176 176 F F F 5 100-119 99 99 99 176 176 176 T T F <--- 6 120-139 98 98 98 170 170 170 F F F 7 140-159 98 98 98 172 172 172 F F F 8 160-179 100 100 99 178 178 176 F F F <--- 9 180-199 100 100 97 178 178 168 F F F <--- 10 200-219 100 100 95 178 178 163 F F F <--- 11 220-239 100 100 90 178 178 152 F F F <--- 12 240-259 92 92 92 155 155 155 T T F <--- 13 260-279 94 94 94 162 162 162 T T F <--- 14 280-299 96 96 96 166 166 166 T T F <--- 15 300-319 98 98 98 174 174 174 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 100 100 94 177 177 159 F F F <--- 18 360-379 98 98 98 171 171 171 T T F <--- 19 380-399 94 94 94 162 162 162 F F F 20 400-419 96 96 96 165 165 165 F F F 21 420-439 94 94 94 162 162 162 F F F 22 440-459 97 97 97 168 168 168 F F F 23 460-479 97 97 97 168 168 168 F F F 24 480-499 93 93 93 156 156 156 F F F 25 500-519 100 100 96 177 177 165 F F T <--- 26 520-539 98 98 98 170 170 170 F F F 27 540-559 96 96 96 165 165 165 F F F 28 560-579 99 99 99 176 176 176 F F F 29 580-599 96 96 96 166 166 166 F F F Max gen: 100, Max flush: 178 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 101 101 101 179 179 179 F F F 1 020-039 100 100 100 178 178 178 F F F 2 040-059 97 97 97 168 168 168 F F F 3 060-079 100 100 100 177 177 177 F F F 4 080-099 99 99 99 176 176 176 F F F 5 100-119 101 101 101 179 179 179 F F F 6 120-139 98 98 98 170 170 170 F F F 7 140-159 98 98 98 172 172 172 F F F 8 160-179 100 100 100 178 178 178 F F F 9 180-199 100 100 100 178 178 178 F F F 10 200-219 100 100 100 178 178 178 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 101 101 101 179 179 179 F F F 13 260-279 101 101 101 179 179 179 F F F 14 280-299 101 101 101 179 179 179 F F F 15 300-319 98 98 98 174 174 174 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 100 100 100 177 177 177 F F F 18 360-379 101 101 101 179 179 179 F F F 19 380-399 94 94 94 162 162 162 F F F 20 400-419 96 96 96 165 165 165 F F F 21 420-439 94 94 94 162 162 162 F F F 22 440-459 97 97 97 168 168 168 F F F 23 460-479 97 97 97 168 168 168 F F F 24 480-499 93 93 93 156 156 156 F F F 25 500-519 100 100 100 177 177 177 F F F 26 520-539 98 98 98 170 170 170 F F F 27 540-559 96 96 96 165 165 165 F F F 28 560-579 99 99 99 176 176 176 F F F 29 580-599 96 96 96 166 166 166 F F F Max gen: 101, Max flush: 179 Begin loop 51 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.274434835Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:43.28505704Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.285109537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.295313774Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.295634753Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.295674668Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29568684Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.295704539Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613} {"msg":"Crucible 6acacf0c-9ee3-4a5e-82e2-b3d3fece53f0 has session id: 11363a9e-f366-4dc3-ae5c-826c8c7a76eb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296043666Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Upstairs opts: Upstairs UUID: 6acacf0c-9ee3-4a5e-82e2-b3d3fece53f0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296071388Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Crucible stats registered with UUID: 6acacf0c-9ee3-4a5e-82e2-b3d3fece53f0","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296086139Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:43.296099121Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296267575Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"io task","client":"1","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29629189Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"io task","client":"0","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296309509Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"io task","client":"2","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296424776Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"task":"crutest"} {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296439947Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613} {"msg":"6acacf0c-9ee3-4a5e-82e2-b3d3fece53f0 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296461733Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296474595Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"client":"0","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296488576Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"client":"1","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296502407Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"client":"2","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296516249Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"io task","client":"2","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29653079Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"io task","client":"0","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296545091Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"io task","client":"1","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:36821)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296602435Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"io task","client":"2","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:62852)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296719361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"io task","client":"1","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:33331)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.296742247Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"io task","client":"0","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.297359099Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"client":"1","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.297387361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"client":"2","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.297463054Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"client":"0","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.297728427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.297775028Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"[0]R flush_numbers[0..12]: [179, 178, 168, 177, 176, 179, 170, 172, 178, 178, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.297831342Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"[0]R generation[0..12]: [101, 100, 97, 100, 99, 101, 98, 98, 100, 100, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.297849771Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29786669Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"[1]R flush_numbers[0..12]: [179, 178, 168, 177, 176, 179, 170, 172, 178, 178, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29788252Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"[1]R generation[0..12]: [101, 100, 97, 100, 99, 101, 98, 98, 100, 100, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29789817Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29791456Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"[2]R flush_numbers[0..12]: [179, 178, 168, 177, 176, 179, 170, 172, 178, 178, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29793033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"[2]R generation[0..12]: [101, 100, 97, 100, 99, 101, 98, 98, 100, 100, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29794582Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29796226Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Max found gen is 102Wait for a query_work_queue command to finish before sending IO ","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.297981978Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.297996878Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Next flush: 180","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29801071Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29802646Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"":"downstairs","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.298040661Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.298053393Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"client":"0","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:04:43.298067524Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"client":"1","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.298084563Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"client":"2","":"downstairs","session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.298098774Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.298112206Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"6acacf0c-9ee3-4a5e-82e2-b3d3fece53f0 is now active with session: 11363a9e-f366-4dc3-ae5c-826c8c7a76eb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.298127066Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.298140088Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613,"session_id":"11363a9e-f366-4dc3-ae5c-826c8c7a76eb"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.29815346Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613} {"msg":"Activated sub_volume 6acacf0c-9ee3-4a5e-82e2-b3d3fece53f0","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:43.298164913Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 156 len 20480 data: 7 8 9 8 6 02/30 Read block 541 len 40960 03/30 Write block 379 len 16384 data: 7 7 6 5 04/30 Write block 348 len 12288 data: 10 11 12 05/30 Read block 20 len 36864 06/30 Read block 542 len 40960 07/30 Write block 580 len 20480 data: 12 12 13 14 13 08/30 Write block 252 len 28672 data: 11 11 11 10 10 9 9 09/30 Write block 284 len 4096 data: 9 10/30 Read block 524 len 32768 11/30 Flush 12/30 Flush 13/30 Write block 177 len 24576 data: 11 12 10 8 7 6 14/30 Write block 3 len 16384 data: 8 7 7 6 15/30 Write block 459 len 36864 data: 10 8 6 5 4 4 4 4 4 16/30 Write block 449 len 20480 data: 9 7 8 7 9 17/30 Write block 14 len 8192 data: 7 8 18/30 Read block 291 len 4096 19/30 Write block 583 len 20480 data: 15 14 10 9 8 20/30 Read block 205 len 24576 21/30 Flush 22/30 Write block 248 len 20480 data: 17 16 17 14 12 23/30 Write block 453 len 8192 data: 10 8 24/30 Flush 25/30 Write block 529 len 40960 data: 11 10 11 10 10 9 7 6 4 5 26/30 Read block 342 len 24576 27/30 Write block 163 len 32768 data: 8 6 7 7 8 8 7 7 28/30 Read block 216 len 40960 29/30 Read block 98 len 36864 30/30 Read block 588 len 16384 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Read 10 Done Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Write 2 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Read 4 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 0 0 24 Done 6 30 30 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:43.31023126Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1613} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 101 102 102 179 182 182 F F F <--- 1 020-039 100 100 100 178 178 178 F F F 2 040-059 97 97 97 168 168 168 F F F 3 060-079 100 100 100 177 177 177 F F F 4 080-099 99 99 99 176 176 176 F F F 5 100-119 101 101 101 179 179 179 F F F 6 120-139 98 98 98 170 170 170 F F F 7 140-159 98 102 102 172 180 180 T F F <--- 8 160-179 100 102 102 178 182 182 T T T <--- 9 180-199 100 102 102 178 182 182 F F F <--- 10 200-219 100 100 100 178 178 178 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 101 102 102 179 183 183 F F F <--- 13 260-279 101 101 101 179 179 179 F F F 14 280-299 101 102 102 179 180 180 F F F <--- 15 300-319 98 98 98 174 174 174 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 100 102 102 177 180 180 T F F <--- 18 360-379 101 102 102 179 180 180 T F F <--- 19 380-399 94 102 102 162 180 180 T F F <--- 20 400-419 96 96 96 165 165 165 F F F 21 420-439 94 94 94 162 162 162 F F F 22 440-459 97 102 102 168 183 183 F F F <--- 23 460-479 97 102 102 168 182 182 F F F <--- 24 480-499 93 93 93 156 156 156 F F F 25 500-519 100 100 100 177 177 177 F F F 26 520-539 98 98 98 170 170 170 F T T <--- 27 540-559 96 96 96 165 165 165 F F F 28 560-579 99 99 99 176 176 176 F F F 29 580-599 96 102 102 166 182 182 F F F <--- Max gen: 102, Max flush: 183 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 182 182 182 F F F 1 020-039 100 100 100 178 178 178 F F F 2 040-059 97 97 97 168 168 168 F F F 3 060-079 100 100 100 177 177 177 F F F 4 080-099 99 99 99 176 176 176 F F F 5 100-119 101 101 101 179 179 179 F F F 6 120-139 98 98 98 170 170 170 F F F 7 140-159 102 102 102 180 180 180 F F F 8 160-179 103 103 103 184 184 184 F F F 9 180-199 102 102 102 182 182 182 F F F 10 200-219 100 100 100 178 178 178 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 102 102 102 183 183 183 F F F 13 260-279 101 101 101 179 179 179 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 98 98 98 174 174 174 F F F 16 320-339 98 98 98 174 174 174 F F F 17 340-359 102 102 102 180 180 180 F F F 18 360-379 102 102 102 180 180 180 F F F 19 380-399 102 102 102 180 180 180 F F F 20 400-419 96 96 96 165 165 165 F F F 21 420-439 94 94 94 162 162 162 F F F 22 440-459 102 102 102 183 183 183 F F F 23 460-479 102 102 102 182 182 182 F F F 24 480-499 93 93 93 156 156 156 F F F 25 500-519 100 100 100 177 177 177 F F F 26 520-539 103 103 103 184 184 184 F F F 27 540-559 96 96 96 165 165 165 F F F 28 560-579 99 99 99 176 176 176 F F F 29 580-599 102 102 102 182 182 182 F F F Max gen: 103, Max flush: 184 Begin loop 52 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.642761847Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:47.653436029Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.653490875Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.663792231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664082059Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664121784Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664136855Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664154843Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621} {"msg":"Crucible 15d95497-9789-4db7-8ac2-ef4308c1a7ec has session id: 6e6ec507-32bb-4d8b-bbca-5aa5966d7de5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664512399Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Upstairs opts: Upstairs UUID: 15d95497-9789-4db7-8ac2-ef4308c1a7ec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664547317Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Crucible stats registered with UUID: 15d95497-9789-4db7-8ac2-ef4308c1a7ec","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664563307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:47.664576768Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664717959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"io task","client":"1","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664745112Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"io task","client":"0","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664760103Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"io task","client":"2","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664887513Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"task":"crutest"} {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664902394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621} {"msg":"15d95497-9789-4db7-8ac2-ef4308c1a7ec active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.6649244Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664937372Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"client":"0","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664951573Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"client":"1","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664967682Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"client":"2","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664981564Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"io task","client":"2","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.664995845Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"io task","client":"0","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.665010036Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"io task","client":"1","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"ds_connection connected from Ok(127.0.0.1:54123)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.665025266Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"io task","client":"2","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"ds_connection connected from Ok(127.0.0.1:56773)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.665046273Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"io task","client":"1","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"ds_connection connected from Ok(127.0.0.1:49650)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.665166368Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"io task","client":"0","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.665857493Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"client":"0","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.66591046Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"client":"1","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.665928299Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"client":"2","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666234207Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666284905Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"[0]R flush_numbers[0..12]: [182, 178, 168, 177, 176, 179, 170, 180, 184, 182, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666322791Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"[0]R generation[0..12]: [102, 100, 97, 100, 99, 101, 98, 102, 103, 102, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.66633974Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.66635625Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"[1]R flush_numbers[0..12]: [182, 178, 168, 177, 176, 179, 170, 180, 184, 182, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666374848Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"[1]R generation[0..12]: [102, 100, 97, 100, 99, 101, 98, 102, 103, 102, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666390688Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666406808Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"[2]R flush_numbers[0..12]: [182, 178, 168, 177, 176, 179, 170, 180, 184, 182, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666422318Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"[2]R generation[0..12]: [102, 100, 97, 100, 99, 101, 98, 102, 103, 102, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666437849Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666454288Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Max found gen is 104","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666473486Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Generation requested: 104 >= found:104","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666487497Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Next flush: 185","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666500879Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666514171Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"":"downstairs","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666527902Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666540404Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"client":"0","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666554185Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"client":"1","":"downstairs","session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666568027Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"client":"2","":"downstairs","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666587574Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666600936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"15d95497-9789-4db7-8ac2-ef4308c1a7ec is now active with session: 6e6ec507-32bb-4d8b-bbca-5aa5966d7de5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666615767Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666628459Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621,"session_id":"6e6ec507-32bb-4d8b-bbca-5aa5966d7de5"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666641671Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621} {"msg":"Activated sub_volume 15d95497-9789-4db7-8ac2-ef4308c1a7ec","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:47.666652993Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 410 len 20480 data: 5 6 6 7 8 02/30 Flush 03/30 Flush 04/30 Write block 82 len 28672 data: 7 8 8 11 11 10 7 05/30 Read block 340 len 4096 06/30 Read block 86 len 20480 07/30 Read block 205 len 36864 08/30 Read block 115 len 4096 09/30 Write block 248 len 36864 data: 18 17 18 15 13 12 12 11 11 10/30 Write block 493 len 12288 data: 10 9 10 11/30 Write block 152 len 36864 data: 5 5 5 7 8 9 10 9 7 12/30 Flush 13/30 Write block 413 len 16384 data: 8 9 6 7 14/30 Write block 447 len 40960 data: 10 8 10 8 9 8 11 9 11 10 15/30 Read block 121 len 24576 16/30 Write block 21 len 28672 data: 9 9 6 5 4 3 6 17/30 Write block 493 len 28672 data: 11 10 11 8 5 6 7 18/30 Read block 358 len 36864 19/30 Read block 444 len 4096 20/30 Read block 333 len 12288 21/30 Write block 361 len 16384 data: 10 10 8 7 22/30 Read block 343 len 36864 23/30 Write block 374 len 20480 data: 5 6 6 5 6 24/30 Read block 289 len 8192 25/30 Write block 329 len 12288 data: 7 8 9 26/30 Read block 499 len 12288 27/30 Write block 454 len 20480 data: 10 12 11 9 9 28/30 Read block 64 len 8192 29/30 Read block 515 len 16384 30/30 Read block 14 len 20480 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Read 5 Sent Done Done false 1006 Acked Read 9 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 6 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Read 5 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 0 29 Done 0 27 28 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1011)) Some(JobId(1011)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:47.679048993Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1621} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 182 182 182 F F F 1 020-039 100 100 100 178 178 178 F T T <--- 2 040-059 97 97 97 168 168 168 F F F 3 060-079 100 100 100 177 177 177 F F F 4 080-099 99 104 104 176 187 187 F F F <--- 5 100-119 101 101 101 179 179 179 F F F 6 120-139 98 98 98 170 170 170 F F F 7 140-159 102 104 104 180 187 187 F F F <--- 8 160-179 103 104 104 184 187 187 F F F <--- 9 180-199 102 102 102 182 182 182 F F F 10 200-219 100 100 100 178 178 178 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 102 104 104 183 187 187 F F F <--- 13 260-279 101 101 101 179 179 179 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 98 98 98 174 174 174 F F F 16 320-339 98 98 98 174 174 174 F T T <--- 17 340-359 102 102 102 180 180 180 F F F 18 360-379 102 102 102 180 180 180 F T T <--- 19 380-399 102 102 102 180 180 180 F F F 20 400-419 104 104 104 185 185 185 F T T <--- 21 420-439 94 94 94 162 162 162 F F F 22 440-459 102 102 102 183 183 183 F T T <--- 23 460-479 102 102 102 182 182 182 F F F 24 480-499 93 104 104 156 187 187 F T T <--- 25 500-519 100 100 100 177 177 177 F F F 26 520-539 103 103 103 184 184 184 F F F 27 540-559 96 96 96 165 165 165 F F F 28 560-579 99 99 99 176 176 176 F F F 29 580-599 102 102 102 182 182 182 F F F Max gen: 104, Max flush: 187 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 182 182 182 F F F 1 020-039 105 105 105 188 188 188 F F F 2 040-059 97 97 97 168 168 168 F F F 3 060-079 100 100 100 177 177 177 F F F 4 080-099 104 104 104 187 187 187 F F F 5 100-119 101 101 101 179 179 179 F F F 6 120-139 98 98 98 170 170 170 F F F 7 140-159 104 104 104 187 187 187 F F F 8 160-179 104 104 104 187 187 187 F F F 9 180-199 102 102 102 182 182 182 F F F 10 200-219 100 100 100 178 178 178 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 101 101 101 179 179 179 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 98 98 98 174 174 174 F F F 16 320-339 105 105 105 188 188 188 F F F 17 340-359 102 102 102 180 180 180 F F F 18 360-379 105 105 105 188 188 188 F F F 19 380-399 102 102 102 180 180 180 F F F 20 400-419 105 105 105 188 188 188 F F F 21 420-439 94 94 94 162 162 162 F F F 22 440-459 105 105 105 188 188 188 F F F 23 460-479 102 102 102 182 182 182 F F F 24 480-499 105 105 105 188 188 188 F F F 25 500-519 100 100 100 177 177 177 F F F 26 520-539 103 103 103 184 184 184 F F F 27 540-559 96 96 96 165 165 165 F F F 28 560-579 99 99 99 176 176 176 F F F 29 580-599 102 102 102 182 182 182 F F F Max gen: 105, Max flush: 188 Begin loop 53 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.003964166Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:52.014579086Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.01463652Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.02485147Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025140499Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025177685Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025189698Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025207177Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629} {"msg":"Crucible 8ecdde90-301e-46d8-a3c5-770d20fc697a has session id: 6cdfa6c1-336f-44d4-b005-158c4942fa88","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025558926Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Upstairs opts: Upstairs UUID: 8ecdde90-301e-46d8-a3c5-770d20fc697a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025593424Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Crucible stats registered with UUID: 8ecdde90-301e-46d8-a3c5-770d20fc697a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025611742Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:52.025625434Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025768884Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"io task","client":"2","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025796866Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"io task","client":"0","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025811687Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"io task","client":"1","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025927704Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"task":"crutest"} {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025945133Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629} {"msg":"8ecdde90-301e-46d8-a3c5-770d20fc697a active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025975624Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.025989415Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"client":"0","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.026003416Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"client":"1","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.026017418Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"client":"2","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.026031259Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"io task","client":"2","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.02604583Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"io task","client":"0","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.026060141Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"io task","client":"1","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"ds_connection connected from Ok(127.0.0.1:61626)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.026093999Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"io task","client":"1","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"ds_connection connected from Ok(127.0.0.1:36882)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.026109919Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"io task","client":"0","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"ds_connection connected from Ok(127.0.0.1:60277)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.026137172Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"io task","client":"2","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.026865105Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"client":"2","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.026893777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"client":"1","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.026910246Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"client":"0","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027240159Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027282722Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"[0]R flush_numbers[0..12]: [182, 188, 168, 177, 187, 179, 170, 187, 187, 182, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027321588Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"[0]R generation[0..12]: [102, 105, 97, 100, 104, 101, 98, 104, 104, 102, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027338527Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027355047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"[1]R flush_numbers[0..12]: [182, 188, 168, 177, 187, 179, 170, 187, 187, 182, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027370967Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"[1]R generation[0..12]: [102, 105, 97, 100, 104, 101, 98, 104, 104, 102, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027387027Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027403357Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"[2]R flush_numbers[0..12]: [182, 188, 168, 177, 187, 179, 170, 187, 187, 182, 178, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027419017Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"[2]R generation[0..12]: [102, 105, 97, 100, 104, 101, 98, 104, 104, 102, 100, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027434687Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027453825Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Max found gen is 106","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027474092Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027488533Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Next flush: 189","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027502025Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027515266Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"":"downstairs","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027529477Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027542019Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"client":"0","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027556241Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"client":"1","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:04:52.027570182Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"client":"2","":"downstairs","session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027587811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027601142Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"8ecdde90-301e-46d8-a3c5-770d20fc697a is now active with session: 6cdfa6c1-336f-44d4-b005-158c4942fa88","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027616053Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027628775Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629,"session_id":"6cdfa6c1-336f-44d4-b005-158c4942fa88"} {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027642206Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629} {"msg":"Activated sub_volume 8ecdde90-301e-46d8-a3c5-770d20fc697a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:52.027653659Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 194 len 28672 data: 8 12 8 9 6 8 8 02/30 Flush 03/30 Write block 11 len 16384 data: 5 6 6 8 04/30 Write block 319 len 28672 data: 7 8 6 6 7 8 9 05/30 Write block 162 len 24576 data: 7 9 7 8 8 9 06/30 Flush 07/30 Flush 08/30 Write block 403 len 12288 data: 7 6 8 09/30 Read block 337 len 4096 10/30 Write block 483 len 12288 data: 10 7 6 11/30 Read block 448 len 12288 12/30 Flush 13/30 Write block 441 len 20480 data: 10 12 13 13 12 14/30 Read block 32 len 32768 15/30 Read block 239 len 32768 16/30 Write block 432 len 24576 data: 9 9 11 11 11 9 17/30 Write block 15 len 4096 data: 9 18/30 Read block 586 len 28672 19/30 Flush 20/30 Write block 262 len 24576 data: 7 9 9 9 9 8 21/30 Flush 22/30 Write block 307 len 28672 data: 11 15 14 14 14 16 14 23/30 Read block 314 len 28672 24/30 Read block 220 len 8192 25/30 Write block 492 len 28672 data: 10 12 11 12 9 6 7 26/30 Read block 487 len 4096 27/30 Write block 172 len 20480 data: 5 5 7 7 10 28/30 Read block 210 len 40960 29/30 Read block 193 len 24576 30/30 Write block 70 len 20480 data: 11 11 9 8 12 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 4 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Write 6 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:52.040062042Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1629} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 106 106 182 193 193 F F F <--- 1 020-039 105 105 105 188 188 188 F F F 2 040-059 97 97 97 168 168 168 F F F 3 060-079 100 100 100 177 177 177 F T T <--- 4 080-099 104 104 104 187 187 187 F F F 5 100-119 101 101 101 179 179 179 F F F 6 120-139 98 98 98 170 170 170 F F F 7 140-159 104 104 104 187 187 187 F F F 8 160-179 104 106 106 187 190 190 F T T <--- 9 180-199 102 106 106 182 189 189 F F F <--- 10 200-219 100 106 106 178 189 189 F F F <--- 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 101 106 106 179 194 194 F F F <--- 14 280-299 102 102 102 180 180 180 F F F 15 300-319 98 106 106 174 190 190 F T T <--- 16 320-339 105 106 106 188 190 190 F F F <--- 17 340-359 102 102 102 180 180 180 F F F 18 360-379 105 105 105 188 188 188 F F F 19 380-399 102 102 102 180 180 180 F F F 20 400-419 105 106 106 188 192 192 F F F <--- 21 420-439 94 106 106 162 193 193 F F F <--- 22 440-459 105 106 106 188 193 193 F F F <--- 23 460-479 102 102 102 182 182 182 F F F 24 480-499 105 106 106 188 192 192 F T T <--- 25 500-519 100 100 100 177 177 177 F F F 26 520-539 103 103 103 184 184 184 F F F 27 540-559 96 96 96 165 165 165 F F F 28 560-579 99 99 99 176 176 176 F F F 29 580-599 102 102 102 182 182 182 F F F Max gen: 106, Max flush: 194 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 106 106 106 193 193 193 F F F 1 020-039 105 105 105 188 188 188 F F F 2 040-059 97 97 97 168 168 168 F F F 3 060-079 107 107 107 195 195 195 F F F 4 080-099 104 104 104 187 187 187 F F F 5 100-119 101 101 101 179 179 179 F F F 6 120-139 98 98 98 170 170 170 F F F 7 140-159 104 104 104 187 187 187 F F F 8 160-179 107 107 107 195 195 195 F F F 9 180-199 106 106 106 189 189 189 F F F 10 200-219 106 106 106 189 189 189 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 106 106 106 194 194 194 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 107 107 107 195 195 195 F F F 16 320-339 106 106 106 190 190 190 F F F 17 340-359 102 102 102 180 180 180 F F F 18 360-379 105 105 105 188 188 188 F F F 19 380-399 102 102 102 180 180 180 F F F 20 400-419 106 106 106 192 192 192 F F F 21 420-439 106 106 106 193 193 193 F F F 22 440-459 106 106 106 193 193 193 F F F 23 460-479 102 102 102 182 182 182 F F F 24 480-499 107 107 107 195 195 195 F F F 25 500-519 100 100 100 177 177 177 F F F 26 520-539 103 103 103 184 184 184 F F F 27 540-559 96 96 96 165 165 165 F F F 28 560-579 99 99 99 176 176 176 F F F 29 580-599 102 102 102 182 182 182 F F F Max gen: 107, Max flush: 195 Begin loop 54 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.365879408Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.377611065Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.377926267Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.377963844Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.377976276Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.377994045Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637} {"msg":"Crucible 58cfce41-22f0-4bca-8d1b-2fb67e204ff7 has session id: 0d8c9f5e-44f8-4379-9fba-767da8709577","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378354448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Upstairs opts: Upstairs UUID: 58cfce41-22f0-4bca-8d1b-2fb67e204ff7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378389116Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Crucible stats registered with UUID: 58cfce41-22f0-4bca-8d1b-2fb67e204ff7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378404797Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:56.378418038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378597006Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"io task","client":"2","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378631364Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"io task","client":"0","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378647004Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"io task","client":"1","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378754907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"task":"crutest"} {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378769298Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637} {"msg":"58cfce41-22f0-4bca-8d1b-2fb67e204ff7 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378794592Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378807484Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"client":"0","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378821335Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"client":"1","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378835236Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"client":"2","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378849147Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"io task","client":"2","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378863478Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"io task","client":"0","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378877819Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"io task","client":"1","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"ds_connection connected from Ok(127.0.0.1:40710)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378963036Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"io task","client":"2","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"ds_connection connected from Ok(127.0.0.1:63209)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.378990169Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"io task","client":"1","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"ds_connection connected from Ok(127.0.0.1:48527)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.379006189Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"io task","client":"0","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.379756727Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"client":"0","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.379793094Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"client":"1","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.379810223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"client":"2","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380132561Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.38018114Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"[0]R flush_numbers[0..12]: [193, 188, 168, 195, 187, 179, 170, 187, 195, 189, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380217487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"[0]R generation[0..12]: [106, 105, 97, 107, 104, 101, 98, 104, 107, 106, 106, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380237605Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380254484Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"[1]R flush_numbers[0..12]: [193, 188, 168, 195, 187, 179, 170, 187, 195, 189, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380270094Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"[1]R generation[0..12]: [106, 105, 97, 107, 104, 101, 98, 104, 107, 106, 106, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380285565Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380301675Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"[2]R flush_numbers[0..12]: [193, 188, 168, 195, 187, 179, 170, 187, 195, 189, 189, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380322961Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"[2]R generation[0..12]: [106, 105, 97, 107, 104, 101, 98, 104, 107, 106, 106, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380338661Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380355101Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380368782Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"Wait for a query_work_queue command to finish before sending IO } {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.38038921Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Next flush: 196","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380402671Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380415833Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"":"downstairs","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380429494Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380442086Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"client":"0","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380457637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"client":"1","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380471788Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"client":"2","":"downstairs","session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:04:56.380486039Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380502268Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"58cfce41-22f0-4bca-8d1b-2fb67e204ff7 is now active with session: 0d8c9f5e-44f8-4379-9fba-767da8709577","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380516959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380530331Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637,"session_id":"0d8c9f5e-44f8-4379-9fba-767da8709577"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380543363Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637} {"msg":"Activated sub_volume 58cfce41-22f0-4bca-8d1b-2fb67e204ff7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:04:56.380554765Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 421 len 16384 data: 9 7 8 7 02/30 Write block 53 len 24576 data: 9 9 8 6 6 6 03/30 Read block 55 len 20480 04/30 Write block 158 len 36864 data: 11 10 8 6 8 10 8 9 9 05/30 Write block 592 len 12288 data: 9 9 5 06/30 Write block 155 len 16384 data: 8 9 10 12 07/30 Write block 541 len 4096 data: 5 08/30 Flush 09/30 Write block 547 len 24576 data: 8 9 8 9 8 7 10/30 Read block 444 len 20480 11/30 Read block 169 len 20480 12/30 Write block 339 len 36864 data: 8 7 7 9 8 7 8 8 9 13/30 Write block 568 len 12288 data: 6 7 5 14/30 Read block 328 len 20480 15/30 Flush 16/30 Flush 17/30 Write block 537 len 20480 data: 5 6 6 6 6 18/30 Write block 201 len 20480 data: 9 8 12 10 12 19/30 Write block 149 len 16384 data: 5 6 5 6 20/30 Flush 21/30 Write block 334 len 20480 data: 7 8 8 8 9 22/30 Read block 54 len 8192 23/30 Write block 157 len 12288 data: 11 13 11 24/30 Read block 335 len 16384 25/30 Read block 246 len 40960 26/30 Write block 105 len 8192 data: 9 10 27/30 Write block 400 len 12288 data: 9 10 10 28/30 Write block 137 len 8192 data: 9 7 29/30 Read block 91 len 16384 30/30 Write block 419 len 24576 data: 9 8 10 8 9 8 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 6 Done Sent Done false 1002 Acked Read 5 Done Sent Done false 1003 Acked Write 9 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Write 4 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Write 4 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Read 10 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:04:56.391480929Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1637} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 106 106 106 193 193 193 F F F 1 020-039 105 105 105 188 188 188 F F F 2 040-059 108 97 108 196 168 196 F F F <--- 3 060-079 107 107 107 195 195 195 F F F 4 080-099 104 104 104 187 187 187 F F F 5 100-119 101 101 101 179 179 179 T F T <--- 6 120-139 98 98 98 170 170 170 T F T <--- 7 140-159 108 104 108 199 187 199 T F T <--- 8 160-179 108 107 108 196 195 196 F F F <--- 9 180-199 106 106 106 189 189 189 F F F 10 200-219 108 106 108 199 189 199 F F F <--- 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 106 106 106 194 194 194 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 107 107 107 195 195 195 F F F 16 320-339 108 106 108 197 190 197 T F T <--- 17 340-359 108 102 108 197 180 197 F F F <--- 18 360-379 105 105 105 188 188 188 F F F 19 380-399 102 102 102 180 180 180 F F F 20 400-419 106 106 106 192 192 192 T F T <--- 21 420-439 108 106 108 196 193 196 T T T <--- 22 440-459 106 106 106 193 193 193 F F F 23 460-479 102 102 102 182 182 182 F F F 24 480-499 107 107 107 195 195 195 F F F 25 500-519 100 100 100 177 177 177 F F F 26 520-539 108 103 108 199 184 199 F F F <--- 27 540-559 108 96 108 199 165 199 F F F <--- 28 560-579 108 99 108 197 176 197 F F F <--- 29 580-599 108 102 108 196 182 196 F F F <--- Max gen: 108, Max flush: 199 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 106 106 106 193 193 193 F F F 1 020-039 105 105 105 188 188 188 F F F 2 040-059 108 108 108 196 196 196 F F F 3 060-079 107 107 107 195 195 195 F F F 4 080-099 104 104 104 187 187 187 F F F 5 100-119 109 109 109 200 200 200 F F F 6 120-139 109 109 109 200 200 200 F F F 7 140-159 109 109 109 200 200 200 F F F 8 160-179 108 108 108 196 196 196 F F F 9 180-199 106 106 106 189 189 189 F F F 10 200-219 108 108 108 199 199 199 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 106 106 106 194 194 194 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 107 107 107 195 195 195 F F F 16 320-339 109 109 109 200 200 200 F F F 17 340-359 108 108 108 197 197 197 F F F 18 360-379 105 105 105 188 188 188 F F F 19 380-399 102 102 102 180 180 180 F F F 20 400-419 109 109 109 200 200 200 F F F 21 420-439 109 109 109 200 200 200 F F F 22 440-459 106 106 106 193 193 193 F F F 23 460-479 102 102 102 182 182 182 F F F 24 480-499 107 107 107 195 195 195 F F F 25 500-519 100 100 100 177 177 177 F F F 26 520-539 108 108 108 199 199 199 F F F 27 540-559 108 108 108 199 199 199 F F F 28 560-579 108 108 108 197 197 197 F F F 29 580-599 108 108 108 196 196 196 F F F Max gen: 109, Max flush: 200 Begin loop 55 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.715891409Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:00.726465024Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.726521719Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.736893531Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.737183998Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.737223463Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.737235596Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.737253335Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645} {"msg":"Crucible 3f5cddc1-b796-48a4-b171-ce9bf5689438 has session id: 4e141443-d377-4ad1-a625-9466393efca1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.737611919Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Upstairs opts: Upstairs UUID: 3f5cddc1-b796-48a4-b171-ce9bf5689438, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.737646248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Crucible stats registered with UUID: 3f5cddc1-b796-48a4-b171-ce9bf5689438","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.737662028Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:00.73767521Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.737821977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"io task","client":"1","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.73785016Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"io task","client":"0","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.73786475Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"io task","client":"2","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.73799207Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"task":"crutest"} {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738006242Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645} {"msg":"3f5cddc1-b796-48a4-b171-ce9bf5689438 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738036912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738050544Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"client":"0","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738064805Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"client":"1","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738078646Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"client":"2","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738092407Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"io task","client":"2","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738106998Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"io task","client":"0","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738123248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"io task","client":"1","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"ds_connection connected from Ok(127.0.0.1:58104)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738138139Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"io task","client":"2","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"ds_connection connected from Ok(127.0.0.1:49437)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738153339Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"io task","client":"1","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"ds_connection connected from Ok(127.0.0.1:45553)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.73816811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"io task","client":"0","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738895463Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"client":"1","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738945491Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"client":"0","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.738984867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"client":"2","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739254217Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739298579Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"[0]R flush_numbers[0..12]: [193, 188, 196, 195, 187, 200, 200, 200, 196, 189, 199, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739362589Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"[0]R generation[0..12]: [106, 105, 108, 107, 104, 109, 109, 109, 108, 106, 108, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739379798Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739396178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"[1]R flush_numbers[0..12]: [193, 188, 196, 195, 187, 200, 200, 200, 196, 189, 199, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739412028Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"[1]R generation[0..12]: [106, 105, 108, 107, 104, 109, 109, 109, 108, 106, 108, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739427638Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739446506Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"[2]R flush_numbers[0..12]: [193, 188, 196, 195, 187, 200, 200, 200, 196, 189, 199, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739462546Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"[2]R generation[0..12]: [106, 105, 108, 107, 104, 109, 109, 109, 108, 106, 108, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739477997Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739494376Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739507848Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"Wait for a query_work_queue command to finish before sending IO msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739526806Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Next flush: 201","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739540188Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739553379Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"":"downstairs","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.73956771Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739580362Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"client":"0","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739594324Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"client":"1","":"downstairs","session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739608105Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739624325Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739637576Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"3f5cddc1-b796-48a4-b171-ce9bf5689438 is now active with session: 4e141443-d377-4ad1-a625-9466393efca1","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739652517Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739666578Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645,"session_id":"4e141443-d377-4ad1-a625-9466393efca1"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.73967973Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645} {"msg":"Activated sub_volume 3f5cddc1-b796-48a4-b171-ce9bf5689438","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:00.739691183Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 213 len 28672 data: 12 10 11 12 13 8 8 02/30 Read block 476 len 24576 03/30 Read block 238 len 16384 04/30 Read block 518 len 12288 05/30 Read block 495 len 36864 06/30 Read block 468 len 20480 07/30 Read block 208 len 4096 08/30 Flush 09/30 Write block 505 len 16384 data: 8 10 9 10 10/30 Write block 371 len 40960 data: 8 7 7 6 7 7 6 7 8 8 11/30 Read block 568 len 36864 12/30 Write block 212 len 20480 data: 10 13 11 12 13 13/30 Read block 374 len 24576 14/30 Write block 328 len 36864 data: 10 8 9 10 6 5 8 9 9 15/30 Read block 361 len 16384 16/30 Read block 147 len 32768 17/30 Read block 258 len 28672 18/30 Read block 485 len 12288 19/30 Write block 262 len 32768 data: 8 10 10 10 10 9 9 10 20/30 Read block 428 len 28672 21/30 Read block 324 len 36864 22/30 Write block 168 len 8192 data: 9 8 23/30 Read block 381 len 28672 24/30 Write block 499 len 20480 data: 8 6 6 5 7 25/30 Read block 434 len 4096 26/30 Read block 429 len 20480 27/30 Write block 414 len 40960 data: 10 7 8 5 8 10 9 11 9 10 28/30 Write block 374 len 12288 data: 7 8 8 29/30 Read block 37 len 40960 30/30 Write block 17 len 36864 data: 8 9 11 10 10 10 7 6 5 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Read 5 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Read 6 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 1 1 27 Done 5 29 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1007)) Some(JobId(1007)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:00.753075692Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1645} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 106 106 106 193 193 193 F T T <--- 1 020-039 105 105 105 188 188 188 F T T <--- 2 040-059 108 108 108 196 196 196 F F F 3 060-079 107 107 107 195 195 195 F F F 4 080-099 104 104 104 187 187 187 F F F 5 100-119 109 109 109 200 200 200 F F F 6 120-139 109 109 109 200 200 200 F F F 7 140-159 109 109 109 200 200 200 F F F 8 160-179 108 108 108 196 196 196 F T T <--- 9 180-199 106 106 106 189 189 189 F F F 10 200-219 108 110 110 199 201 201 T T T <--- 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 106 106 106 194 194 194 F T T <--- 14 280-299 102 102 102 180 180 180 F F F 15 300-319 107 107 107 195 195 195 F F F 16 320-339 109 109 109 200 200 200 F T T <--- 17 340-359 108 108 108 197 197 197 F F F 18 360-379 105 105 105 188 188 188 F T T <--- 19 380-399 102 102 102 180 180 180 F T T <--- 20 400-419 109 109 109 200 200 200 F T T <--- 21 420-439 109 109 109 200 200 200 F T T <--- 22 440-459 106 106 106 193 193 193 F F F 23 460-479 102 102 102 182 182 182 F F F 24 480-499 107 107 107 195 195 195 F T T <--- 25 500-519 100 100 100 177 177 177 F T T <--- 26 520-539 108 108 108 199 199 199 F F F 27 540-559 108 108 108 199 199 199 F F F 28 560-579 108 108 108 197 197 197 F F F 29 580-599 108 108 108 196 196 196 F F F Max gen: 110, Max flush: 201 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 202 202 202 F F F 1 020-039 111 111 111 202 202 202 F F F 2 040-059 108 108 108 196 196 196 F F F 3 060-079 107 107 107 195 195 195 F F F 4 080-099 104 104 104 187 187 187 F F F 5 100-119 109 109 109 200 200 200 F F F 6 120-139 109 109 109 200 200 200 F F F 7 140-159 109 109 109 200 200 200 F F F 8 160-179 111 111 111 202 202 202 F F F 9 180-199 106 106 106 189 189 189 F F F 10 200-219 111 111 111 202 202 202 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 111 111 111 202 202 202 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 107 107 107 195 195 195 F F F 16 320-339 111 111 111 202 202 202 F F F 17 340-359 108 108 108 197 197 197 F F F 18 360-379 111 111 111 202 202 202 F F F 19 380-399 111 111 111 202 202 202 F F F 20 400-419 111 111 111 202 202 202 F F F 21 420-439 111 111 111 202 202 202 F F F 22 440-459 106 106 106 193 193 193 F F F 23 460-479 102 102 102 182 182 182 F F F 24 480-499 111 111 111 202 202 202 F F F 25 500-519 111 111 111 202 202 202 F F F 26 520-539 108 108 108 199 199 199 F F F 27 540-559 108 108 108 199 199 199 F F F 28 560-579 108 108 108 197 197 197 F F F 29 580-599 108 108 108 196 196 196 F F F Max gen: 111, Max flush: 202 Begin loop 56 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.088502568Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.10030707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.100618464Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.100655981Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.100668263Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.100686192Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653} {"msg":"Crucible 7d539701-3691-40ad-a197-f0c5af9bcdea has session id: 7155f0b2-9f5c-4724-8169-e6ba353a8e33","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101033254Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Upstairs opts: Upstairs UUID: 7d539701-3691-40ad-a197-f0c5af9bcdea, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101069291Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Crucible stats registered with UUID: 7d539701-3691-40ad-a197-f0c5af9bcdea","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101085371Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:05.101099042Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101252946Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"io task","client":"2","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101290842Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"io task","client":"0","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101306872Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"io task","client":"1","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101380765Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"task":"crutest"} {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101394976Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653} {"msg":"7d539701-3691-40ad-a197-f0c5af9bcdea active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101416443Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101429485Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"client":"0","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101443406Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"client":"1","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101457487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"client":"2","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101471219Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"io task","client":"2","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101493085Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"io task","client":"0","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101507516Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"io task","client":"1","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"ds_connection connected from Ok(127.0.0.1:43531)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101545062Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"io task","client":"1","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"ds_connection connected from Ok(127.0.0.1:46298)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101560712Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"io task","client":"2","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"ds_connection connected from Ok(127.0.0.1:47493)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.101575813Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"io task","client":"0","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102369934Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"client":"2","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102396907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"client":"1","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102420232Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"client":"0","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102721663Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102765835Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"[0]R flush_numbers[0..12]: [202, 202, 196, 195, 187, 200, 200, 200, 202, 189, 202, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102846394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"[0]R generation[0..12]: [111, 111, 108, 107, 104, 109, 109, 109, 111, 106, 111, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102863504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102880203Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"[1]R flush_numbers[0..12]: [202, 202, 196, 195, 187, 200, 200, 200, 202, 189, 202, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102895883Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"[1]R generation[0..12]: [111, 111, 108, 107, 104, 109, 109, 109, 111, 106, 111, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102911603Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102927663Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"[2]R flush_numbers[0..12]: [202, 202, 196, 195, 187, 200, 200, 200, 202, 189, 202, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102943473Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"[2]R generation[0..12]: [111, 111, 108, 107, 104, 109, 109, 109, 111, 106, 111, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102959193Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102975573Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Max found gen is 112","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.102988935Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Generation requested: 112 >= found:112","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103003056Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Next flush: 203","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103025512Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103038943Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"":"downstairs","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103053304Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103066196Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"client":"0","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103080367Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"client":"1","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103094468Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"client":"2","":"downstairs","session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103108809Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:05:05.103122241Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"7d539701-3691-40ad-a197-f0c5af9bcdea is now active with session: 7155f0b2-9f5c-4724-8169-e6ba353a8e33","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.10314008Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103152752Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653,"session_id":"7155f0b2-9f5c-4724-8169-e6ba353a8e33"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103166393Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653} {"msg":"Activated sub_volume 7d539701-3691-40ad-a197-f0c5af9bcdea","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:05.103177736Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 442 len 8192 data: 13 14 03/30 Read block 528 len 20480 04/30 Read block 223 len 40960 05/30 Read block 366 len 8192 06/30 Read block 549 len 4096 07/30 Write block 487 len 8192 data: 7 7 08/30 Write block 204 len 40960 data: 11 13 10 10 13 12 10 9 11 14 09/30 Read block 255 len 12288 10/30 Write block 264 len 20480 data: 11 11 11 10 10 11/30 Write block 522 len 24576 data: 8 8 8 10 11 12 12/30 Write block 63 len 36864 data: 8 8 9 9 7 9 10 12 12 13/30 Write block 58 len 24576 data: 7 9 11 10 9 9 14/30 Read block 329 len 8192 15/30 Read block 331 len 8192 16/30 Write block 380 len 8192 data: 9 7 17/30 Write block 367 len 24576 data: 12 11 10 9 9 8 18/30 Write block 148 len 20480 data: 7 6 7 6 7 19/30 Write block 276 len 16384 data: 9 6 6 7 20/30 Flush 21/30 Write block 115 len 16384 data: 4 4 5 4 22/30 Write block 60 len 12288 data: 12 11 10 23/30 Read block 148 len 20480 24/30 Read block 539 len 8192 25/30 Read block 219 len 8192 26/30 Write block 334 len 12288 data: 9 10 10 27/30 Read block 372 len 40960 28/30 Read block 228 len 8192 29/30 Write block 378 len 36864 data: 8 9 10 8 6 6 5 4 7 30/30 Write block 213 len 12288 data: 15 12 13 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1020 Acked Write 4 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 10 2 14 Done 8 0 8 16 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: 1019 1018 1017 1016 1015 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:05.114236017Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1653} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 202 202 202 F F F 1 020-039 111 111 111 202 202 202 F F F 2 040-059 112 112 112 204 204 204 F F F 3 060-079 112 112 112 204 204 204 T F T <--- 4 080-099 104 104 104 187 187 187 F F F 5 100-119 109 109 109 200 200 200 T F T <--- 6 120-139 109 109 109 200 200 200 F F F 7 140-159 112 112 112 204 204 204 F F F 8 160-179 111 111 111 202 202 202 F F F 9 180-199 106 106 106 189 189 189 F F F 10 200-219 112 112 112 204 204 204 T F T <--- 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 112 112 112 204 204 204 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 107 107 107 195 195 195 F F F 16 320-339 111 111 111 202 202 202 T F T <--- 17 340-359 108 108 108 197 197 197 F F F 18 360-379 112 112 112 204 204 204 T F T <--- 19 380-399 112 112 112 204 204 204 T F T <--- 20 400-419 111 111 111 202 202 202 F F F 21 420-439 111 111 111 202 202 202 F F F 22 440-459 112 112 112 204 204 204 F F F 23 460-479 102 102 102 182 182 182 F F F 24 480-499 112 112 112 204 204 204 F F F 25 500-519 111 111 111 202 202 202 F F F 26 520-539 112 112 112 204 204 204 F F F 27 540-559 108 108 108 199 199 199 F F F 28 560-579 108 108 108 197 197 197 F F F 29 580-599 108 108 108 196 196 196 F F F Max gen: 112, Max flush: 204 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 202 202 202 F F F 1 020-039 111 111 111 202 202 202 F F F 2 040-059 112 112 112 204 204 204 F F F 3 060-079 113 113 113 205 205 205 F F F 4 080-099 104 104 104 187 187 187 F F F 5 100-119 113 113 113 205 205 205 F F F 6 120-139 109 109 109 200 200 200 F F F 7 140-159 112 112 112 204 204 204 F F F 8 160-179 111 111 111 202 202 202 F F F 9 180-199 106 106 106 189 189 189 F F F 10 200-219 113 113 113 205 205 205 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 112 112 112 204 204 204 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 107 107 107 195 195 195 F F F 16 320-339 113 113 113 205 205 205 F F F 17 340-359 108 108 108 197 197 197 F F F 18 360-379 113 113 113 205 205 205 F F F 19 380-399 113 113 113 205 205 205 F F F 20 400-419 111 111 111 202 202 202 F F F 21 420-439 111 111 111 202 202 202 F F F 22 440-459 112 112 112 204 204 204 F F F 23 460-479 102 102 102 182 182 182 F F F 24 480-499 112 112 112 204 204 204 F F F 25 500-519 111 111 111 202 202 202 F F F 26 520-539 112 112 112 204 204 204 F F F 27 540-559 108 108 108 199 199 199 F F F 28 560-579 108 108 108 197 197 197 F F F 29 580-599 108 108 108 196 196 196 F F F Max gen: 113, Max flush: 205 Begin loop 57 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.448992235Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.460650329Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.460976034Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461017808Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46103069Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461048749Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661} {"msg":"Crucible a3322e3f-86be-4022-a737-4c7c94f4fa2f has session id: c8630ad1-5754-4f6e-99c7-c678802ff5cf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461427011Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Upstairs opts: Upstairs UUID: a3322e3f-86be-4022-a737-4c7c94f4fa2f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461455903Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Crucible stats registered with UUID: a3322e3f-86be-4022-a737-4c7c94f4fa2f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461473662Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:09.461487053Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46165062Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"io task","client":"1","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461685748Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"io task","client":"0","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461701339Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"io task","client":"2","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461814158Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"task":"crutest"} {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461828529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661} {"msg":"a3322e3f-86be-4022-a737-4c7c94f4fa2f active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461844958Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46185754Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"client":"0","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461871831Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"client":"1","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461885643Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"client":"2","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461933543Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"io task","client":"2","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461958097Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"io task","client":"0","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461973198Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"io task","client":"1","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"ds_connection connected from Ok(127.0.0.1:39408)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.461988148Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"io task","client":"2","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"ds_connection connected from Ok(127.0.0.1:64616)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.462027104Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"io task","client":"0","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"ds_connection connected from Ok(127.0.0.1:40244)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46204874Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"io task","client":"1","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46273316Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"client":"0","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.462761932Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"client":"1","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.462786976Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"client":"2","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463104337Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463150448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"[0]R flush_numbers[0..12]: [202, 202, 204, 205, 187, 205, 200, 204, 202, 189, 205, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463224531Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"[0]R generation[0..12]: [111, 111, 112, 113, 104, 113, 109, 112, 111, 106, 113, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46324199Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46325879Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"[1]R flush_numbers[0..12]: [202, 202, 204, 205, 187, 205, 200, 204, 202, 189, 205, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46327475Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"[1]R generation[0..12]: [111, 111, 112, 113, 104, 113, 109, 112, 111, 106, 113, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46328997Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46330611Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"[2]R flush_numbers[0..12]: [202, 202, 204, 205, 187, 205, 200, 204, 202, 189, 205, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.46332165Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"[2]R generation[0..12]: [111, 111, 112, 113, 104, 113, 109, 112, 111, 106, 113, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463339949Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463356269Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:05:09.46336966Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463389388Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Next flush: 206","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463402689Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463416011Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"":"downstairs","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463430062Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463442814Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"client":"0","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463456716Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"client":"1","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463470747Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"client":"2","":"downstairs","session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:05:09.463484718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463499129Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"a3322e3f-86be-4022-a737-4c7c94f4fa2f is now active with session: c8630ad1-5754-4f6e-99c7-c678802ff5cf","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463514229Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463527221Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661,"session_id":"c8630ad1-5754-4f6e-99c7-c678802ff5cf"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463540593Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661} {"msg":"Activated sub_volume a3322e3f-86be-4022-a737-4c7c94f4fa2f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:09.463552076Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 133 len 28672 data: 7 10 11 10 10 8 8 02/30 Write block 575 len 24576 data: 9 10 12 12 11 13 03/30 Write block 364 len 20480 data: 8 7 10 13 12 04/30 Read block 434 len 32768 05/30 Read block 185 len 16384 06/30 Read block 505 len 16384 07/30 Read block 115 len 40960 08/30 Read block 36 len 28672 09/30 Write block 321 len 4096 data: 7 10/30 Read block 246 len 4096 11/30 Read block 414 len 8192 12/30 Write block 501 len 28672 data: 7 6 8 7 9 11 10 13/30 Flush 14/30 Write block 37 len 8192 data: 9 7 15/30 Read block 359 len 12288 16/30 Read block 247 len 28672 17/30 Write block 519 len 4096 data: 10 18/30 Read block 330 len 36864 19/30 Flush 20/30 Write block 464 len 20480 data: 5 5 5 5 5 21/30 Write block 152 len 32768 data: 8 6 6 9 10 12 14 12 22/30 Read block 478 len 8192 23/30 Write block 588 len 12288 data: 7 9 8 24/30 Read block 202 len 4096 25/30 Write block 66 len 16384 data: 10 8 10 11 26/30 Read block 397 len 4096 27/30 Read block 455 len 12288 28/30 Write block 175 len 24576 data: 8 11 12 13 11 9 29/30 Write block 300 len 4096 data: 6 30/30 Read block 37 len 36864 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Write 5 Done Done Sent false 1003 Acked Read 8 Done Done Sent false 1004 Acked Read 4 Done Done Sent false 1005 Acked Read 4 Done Done Sent false 1006 Acked Read 10 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Read 1 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Write 1 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 5 Done Done Sent false 1020 Acked Write 8 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Write 6 Done Done Sent false 1028 Acked Write 1 Done Done Sent false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 28 29 Done 29 30 2 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:09.474866495Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1661} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 202 202 202 F F F 1 020-039 114 114 111 207 207 202 F F F <--- 2 040-059 112 112 112 204 204 204 F F F 3 060-079 113 113 113 205 205 205 T T F <--- 4 080-099 104 104 104 187 187 187 F F F 5 100-119 113 113 113 205 205 205 F F F 6 120-139 114 114 109 206 206 200 F F T <--- 7 140-159 112 112 112 204 204 204 T T F <--- 8 160-179 111 111 111 202 202 202 T T F <--- 9 180-199 106 106 106 189 189 189 T T F <--- 10 200-219 113 113 113 205 205 205 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 112 112 112 204 204 204 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 107 107 107 195 195 195 T T F <--- 16 320-339 114 114 113 206 206 205 F F F <--- 17 340-359 108 108 108 197 197 197 F F F 18 360-379 114 114 113 206 206 205 F F F <--- 19 380-399 113 113 113 205 205 205 F F F 20 400-419 111 111 111 202 202 202 F F F 21 420-439 111 111 111 202 202 202 F F F 22 440-459 112 112 112 204 204 204 F F F 23 460-479 102 102 102 182 182 182 T T F <--- 24 480-499 112 112 112 204 204 204 F F F 25 500-519 114 114 111 207 207 202 F F F <--- 26 520-539 112 112 112 204 204 204 F F F 27 540-559 108 108 108 199 199 199 F F F 28 560-579 114 114 108 206 206 197 F F T <--- 29 580-599 114 114 108 206 206 196 T T T <--- Max gen: 114, Max flush: 207 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 57 Verifying data now No change after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 202 202 202 F F F 1 020-039 114 114 114 207 207 207 F F F 2 040-059 112 112 112 204 204 204 F F F 3 060-079 115 115 115 208 208 208 F F F 4 080-099 104 104 104 187 187 187 F F F 5 100-119 113 113 113 205 205 205 F F F 6 120-139 114 114 114 206 206 206 F F F 7 140-159 115 115 115 208 208 208 F F F 8 160-179 115 115 115 208 208 208 F F F 9 180-199 115 115 115 208 208 208 F F F 10 200-219 113 113 113 205 205 205 F F F 11 220-239 100 100 100 178 178 178 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 112 112 112 204 204 204 F F F 14 280-299 102 102 102 180 180 180 F F F 15 300-319 115 115 115 208 208 208 F F F 16 320-339 114 114 114 206 206 206 F F F 17 340-359 108 108 108 197 197 197 F F F 18 360-379 114 114 114 206 206 206 F F F 19 380-399 113 113 113 205 205 205 F F F 20 400-419 111 111 111 202 202 202 F F F 21 420-439 111 111 111 202 202 202 F F F 22 440-459 112 112 112 204 204 204 F F F 23 460-479 115 115 115 208 208 208 F F F 24 480-499 112 112 112 204 204 204 F F F 25 500-519 114 114 114 207 207 207 F F F 26 520-539 112 112 112 204 204 204 F F F 27 540-559 108 108 108 199 199 199 F F F 28 560-579 114 114 114 206 206 206 F F F 29 580-599 115 115 115 208 208 208 F F F Max gen: 115, Max flush: 208 Begin loop 58 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.797217199Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:13.807802377Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.807852825Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.818227705Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.81852265Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.818560087Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.818572739Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.818590677Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669} {"msg":"Crucible 7d0f923f-a853-4152-921f-71679ef7f5fe has session id: 691ba24f-a796-4e63-969e-7fc5687ffdef","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.818924158Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Upstairs opts: Upstairs UUID: 7d0f923f-a853-4152-921f-71679ef7f5fe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.818958216Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Crucible stats registered with UUID: 7d0f923f-a853-4152-921f-71679ef7f5fe","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.818973747Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:13.818987148Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819136494Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"io task","client":"2","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819164737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"io task","client":"0","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819184284Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"io task","client":"1","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819297413Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"task":"crutest"} {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819312694Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669} {"msg":"7d0f923f-a853-4152-921f-71679ef7f5fe active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.81933418Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819347252Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"client":"0","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819360973Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"client":"1","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819375204Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"client":"2","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819389186Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"io task","client":"2","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819408993Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"io task","client":"0","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819422964Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"io task","client":"1","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"ds_connection connected from Ok(127.0.0.1:49362)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819437665Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"io task","client":"2","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"ds_connection connected from Ok(127.0.0.1:55967)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819505283Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"io task","client":"1","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"ds_connection connected from Ok(127.0.0.1:47485)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.819594107Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"io task","client":"0","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820195209Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"client":"2","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820230687Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"client":"1","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820247626Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"client":"0","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820581396Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820630386Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"[0]R flush_numbers[0..12]: [202, 207, 204, 208, 187, 205, 206, 208, 208, 208, 205, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820674118Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"[0]R generation[0..12]: [111, 114, 112, 115, 104, 113, 114, 115, 115, 115, 113, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820691407Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820708187Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"[1]R flush_numbers[0..12]: [202, 207, 204, 208, 187, 205, 206, 208, 208, 208, 205, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820724307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"[1]R generation[0..12]: [111, 114, 112, 115, 104, 113, 114, 115, 115, 115, 113, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820739877Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820756216Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"[2]R flush_numbers[0..12]: [202, 207, 204, 208, 187, 205, 206, 208, 208, 208, 205, 178]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820772017Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"[2]R generation[0..12]: [111, 114, 112, 115, 104, 113, 114, 115, 115, 115, 113, 100]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820787627Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820803916Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Max found gen is 116","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820817428Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Generation requested: 116 >= found:116","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820831349Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Next flush: 209","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820844511Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820859811Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"":"downstairs","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820879729Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820892661Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"client":"0","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820907012Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"client":"1","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820921183Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"client":"2","":"downstairs","session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820935474Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820948865Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"7d0f923f-a853-4152-921f-71679ef7f5fe is now active with session: 691ba24f-a796-4e63-969e-7fc5687ffdef","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820963876Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820978707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669,"session_id":"691ba24f-a796-4e63-969e-7fc5687ffdef"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.820992258Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669} {"msg":"Activated sub_volume 7d0f923f-a853-4152-921f-71679ef7f5fe","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:13.821003871Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 294 len 32768 data: 9 8 8 7 7 8 7 6 02/30 Read block 370 len 16384 03/30 Write block 459 len 12288 data: 11 9 7 04/30 Write block 497 len 12288 data: 7 8 9 05/30 Write block 39 len 28672 data: 7 6 6 6 5 5 7 06/30 Read block 328 len 12288 07/30 Flush 08/30 Write block 584 len 4096 data: 15 09/30 Read block 424 len 20480 10/30 Read block 536 len 32768 11/30 Write block 329 len 4096 data: 9 12/30 Read block 70 len 4096 13/30 Write block 333 len 40960 data: 6 10 11 11 9 10 9 8 8 10 14/30 Read block 50 len 24576 15/30 Read block 362 len 28672 16/30 Flush 17/30 Write block 446 len 16384 data: 12 11 9 11 18/30 Write block 547 len 16384 data: 9 10 9 10 19/30 Write block 217 len 16384 data: 14 9 9 8 20/30 Write block 405 len 24576 data: 9 8 10 7 4 6 21/30 Write block 321 len 8192 data: 8 7 22/30 Write block 401 len 28672 data: 11 11 8 7 10 9 11 23/30 Flush 24/30 Write block 375 len 32768 data: 9 9 7 9 10 11 9 7 25/30 Read block 342 len 28672 26/30 Flush 27/30 Write block 132 len 4096 data: 6 28/30 Write block 277 len 12288 data: 7 7 8 29/30 Read block 297 len 20480 30/30 Write block 123 len 12288 data: 5 8 6 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Write 3 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Write 7 Done Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Read 5 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Read 7 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 4 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Write 4 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 7 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 1 1 27 Done 5 29 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:13.832232704Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1669} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 202 202 202 F F F 1 020-039 114 116 116 207 209 209 T F F <--- 2 040-059 112 116 116 204 209 209 T F F <--- 3 060-079 115 115 115 208 208 208 F F F 4 080-099 104 104 104 187 187 187 F F F 5 100-119 113 113 113 205 205 205 F F F 6 120-139 114 114 114 206 206 206 F T T <--- 7 140-159 115 115 115 208 208 208 F F F 8 160-179 115 115 115 208 208 208 F F F 9 180-199 115 115 115 208 208 208 F F F 10 200-219 113 116 116 205 211 211 F F F <--- 11 220-239 100 116 116 178 211 211 F F F <--- 12 240-259 104 104 104 187 187 187 F F F 13 260-279 112 112 112 204 204 204 F T T <--- 14 280-299 102 116 116 180 209 209 T F F <--- 15 300-319 115 116 116 208 209 209 T F F <--- 16 320-339 114 116 116 206 211 211 F F F <--- 17 340-359 108 116 116 197 210 210 F F F <--- 18 360-379 114 116 116 206 212 212 F F F <--- 19 380-399 113 116 116 205 212 212 F F F <--- 20 400-419 111 116 116 202 211 211 F F F <--- 21 420-439 111 111 111 202 202 202 F F F 22 440-459 112 116 116 204 211 211 T F F <--- 23 460-479 115 116 116 208 209 209 T F F <--- 24 480-499 112 116 116 204 209 209 T F F <--- 25 500-519 114 114 114 207 207 207 F F F 26 520-539 112 112 112 204 204 204 F F F 27 540-559 108 116 116 199 211 211 F F F <--- 28 560-579 114 114 114 206 206 206 F F F 29 580-599 115 116 116 208 210 210 F F F <--- Max gen: 116, Max flush: 212 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 58 Verifying data now No change after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 202 202 202 F F F 1 020-039 116 116 116 209 209 209 F F F 2 040-059 116 116 116 209 209 209 F F F 3 060-079 115 115 115 208 208 208 F F F 4 080-099 104 104 104 187 187 187 F F F 5 100-119 113 113 113 205 205 205 F F F 6 120-139 117 117 117 213 213 213 F F F 7 140-159 115 115 115 208 208 208 F F F 8 160-179 115 115 115 208 208 208 F F F 9 180-199 115 115 115 208 208 208 F F F 10 200-219 116 116 116 211 211 211 F F F 11 220-239 116 116 116 211 211 211 F F F 12 240-259 104 104 104 187 187 187 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 116 116 116 209 209 209 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 116 116 116 211 211 211 F F F 17 340-359 116 116 116 210 210 210 F F F 18 360-379 116 116 116 212 212 212 F F F 19 380-399 116 116 116 212 212 212 F F F 20 400-419 116 116 116 211 211 211 F F F 21 420-439 111 111 111 202 202 202 F F F 22 440-459 116 116 116 211 211 211 F F F 23 460-479 116 116 116 209 209 209 F F F 24 480-499 116 116 116 209 209 209 F F F 25 500-519 114 114 114 207 207 207 F F F 26 520-539 112 112 112 204 204 204 F F F 27 540-559 116 116 116 211 211 211 F F F 28 560-579 114 114 114 206 206 206 F F F 29 580-599 116 116 116 210 210 210 F F F Max gen: 117, Max flush: 213 Begin loop 59 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.174118841Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.186000684Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.186313908Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.186356121Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.186369173Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.18639026Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677} {"msg":"Crucible 47101950-994f-47c2-9f44-e1c4eaaf7f14 has session id: 6fa3e684-32e4-420e-9774-a398f67b4671","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.186746766Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Upstairs opts: Upstairs UUID: 47101950-994f-47c2-9f44-e1c4eaaf7f14, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.186785571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Crucible stats registered with UUID: 47101950-994f-47c2-9f44-e1c4eaaf7f14","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.186800682Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:18.186813873Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.186980259Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"io task","client":"1","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.187013448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"io task","client":"0","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.187027559Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"io task","client":"2","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.18713718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"task":"crutest"} {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.18715366Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677} {"msg":"47101950-994f-47c2-9f44-e1c4eaaf7f14 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.187171539Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.187184261Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"client":"0","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.187198122Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"client":"1","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.187212123Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"client":"2","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.187225725Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"io task","client":"2","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.187259763Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"io task","client":"0","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.1873919Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"io task","client":"1","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"ds_connection connected from Ok(127.0.0.1:62546)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.187410179Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"io task","client":"0","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"ds_connection connected from Ok(127.0.0.1:59120)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.187425139Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"io task","client":"1","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"ds_connection connected from Ok(127.0.0.1:61995)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.18744009Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"io task","client":"2","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188024043Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"client":"1","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188052205Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"client":"2","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188069005Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"client":"0","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188389154Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188438932Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"[0]R flush_numbers[0..12]: [202, 209, 209, 208, 187, 205, 213, 208, 208, 208, 211, 211]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188477838Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"[0]R generation[0..12]: [111, 116, 116, 115, 104, 113, 117, 115, 115, 115, 116, 116]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188494637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188511207Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"[1]R flush_numbers[0..12]: [202, 209, 209, 208, 187, 205, 213, 208, 208, 208, 211, 211]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188527177Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"[1]R generation[0..12]: [111, 116, 116, 115, 104, 113, 117, 115, 115, 115, 116, 116]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188542837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188561995Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"[2]R flush_numbers[0..12]: [202, 209, 209, 208, 187, 205, 213, 208, 208, 208, 211, 211]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188578175Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"[2]R generation[0..12]: [111, 116, 116, 115, 104, 113, 117, 115, 115, 115, 116, 116]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188593945Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188610325Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Max found gen is 118","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:05:18.188624146Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Generation requested: 118 >= found:118","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188642165Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Next flush: 214","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188655436Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188668828Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"":"downstairs","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188683389Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188696051Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"client":"0","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188710072Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"client":"1","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188724433Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"client":"2","":"downstairs","session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188741032Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188754744Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"47101950-994f-47c2-9f44-e1c4eaaf7f14 is now active with session: 6fa3e684-32e4-420e-9774-a398f67b4671","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188771583Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188784395Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677,"session_id":"6fa3e684-32e4-420e-9774-a398f67b4671"} {"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.188797807Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677} {"msg":"Activated sub_volume 47101950-994f-47c2-9f44-e1c4eaaf7f14","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:18.18880925Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 457 len 12288 data: 10 10 12 02/30 Write block 223 len 32768 data: 8 9 7 6 8 7 3 3 03/30 Write block 228 len 36864 data: 8 4 4 4 6 8 9 12 13 04/30 Flush 05/30 Flush 06/30 Write block 247 len 28672 data: 16 19 18 19 16 14 13 07/30 Read block 185 len 16384 08/30 Read block 571 len 4096 09/30 Write block 12 len 20480 data: 7 7 9 10 7 10/30 Read block 378 len 20480 11/30 Read block 238 len 28672 12/30 Read block 310 len 8192 13/30 Read block 349 len 16384 14/30 Write block 67 len 40960 data: 9 11 12 13 13 10 9 13 10 9 15/30 Read block 172 len 16384 16/30 Read block 427 len 8192 17/30 Write block 169 len 28672 data: 9 8 4 6 6 8 9 18/30 Read block 25 len 32768 19/30 Read block 103 len 16384 20/30 Write block 598 len 8192 data: 3 2 21/30 Flush 22/30 Write block 418 len 4096 data: 9 23/30 Flush 24/30 Write block 78 len 12288 data: 8 10 8 25/30 Read block 535 len 32768 26/30 Read block 568 len 4096 27/30 Write block 232 len 32768 data: 7 9 10 13 14 10 13 13 28/30 Read block 206 len 28672 29/30 Read block 131 len 24576 30/30 Write block 248 len 16384 data: 20 19 20 17 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 8 Done Done Sent false 1002 Acked Write 9 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 3 Done Done Sent false 1024 Acked Read 8 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:18.200640385Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1677} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 111 216 216 202 F F F <--- 1 020-039 116 116 116 209 209 209 F F F 2 040-059 116 116 116 209 209 209 F F F 3 060-079 118 118 115 216 216 208 T T F <--- 4 080-099 104 104 104 187 187 187 T T F <--- 5 100-119 113 113 113 205 205 205 F F F 6 120-139 117 117 117 213 213 213 F F F 7 140-159 115 115 115 208 208 208 F F F 8 160-179 118 118 115 216 216 208 F F F <--- 9 180-199 115 115 115 208 208 208 F F F 10 200-219 116 116 116 211 211 211 F F F 11 220-239 118 118 116 214 214 211 T T F <--- 12 240-259 118 118 104 216 216 187 T T F <--- 13 260-279 117 117 117 213 213 213 F F F 14 280-299 116 116 116 209 209 209 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 116 116 116 211 211 211 F F F 17 340-359 116 116 116 210 210 210 F F F 18 360-379 116 116 116 212 212 212 F F F 19 380-399 116 116 116 212 212 212 F F F 20 400-419 118 118 116 217 217 211 F F F <--- 21 420-439 111 111 111 202 202 202 F F F 22 440-459 118 118 116 214 214 211 F F T <--- 23 460-479 116 116 116 209 209 209 F F F 24 480-499 116 116 116 209 209 209 F F F 25 500-519 114 114 114 207 207 207 F F F 26 520-539 112 112 112 204 204 204 F F F 27 540-559 116 116 116 211 211 211 F F F 28 560-579 114 114 114 206 206 206 F F F 29 580-599 118 118 116 216 216 210 F F F <--- Max gen: 118, Max flush: 217 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 59 Verifying data now No change after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 216 216 216 F F F 1 020-039 116 116 116 209 209 209 F F F 2 040-059 116 116 116 209 209 209 F F F 3 060-079 119 119 119 218 218 218 F F F 4 080-099 119 119 119 218 218 218 F F F 5 100-119 113 113 113 205 205 205 F F F 6 120-139 117 117 117 213 213 213 F F F 7 140-159 115 115 115 208 208 208 F F F 8 160-179 118 118 118 216 216 216 F F F 9 180-199 115 115 115 208 208 208 F F F 10 200-219 116 116 116 211 211 211 F F F 11 220-239 119 119 119 218 218 218 F F F 12 240-259 119 119 119 218 218 218 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 116 116 116 209 209 209 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 116 116 116 211 211 211 F F F 17 340-359 116 116 116 210 210 210 F F F 18 360-379 116 116 116 212 212 212 F F F 19 380-399 116 116 116 212 212 212 F F F 20 400-419 118 118 118 217 217 217 F F F 21 420-439 111 111 111 202 202 202 F F F 22 440-459 118 118 118 214 214 214 F F F 23 460-479 116 116 116 209 209 209 F F F 24 480-499 116 116 116 209 209 209 F F F 25 500-519 114 114 114 207 207 207 F F F 26 520-539 112 112 112 204 204 204 F F F 27 540-559 116 116 116 211 211 211 F F F 28 560-579 114 114 114 206 206 206 F F F 29 580-599 118 118 118 216 216 216 F F F Max gen: 119, Max flush: 218 Begin loop 60 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.511972293Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.523520766Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.523860492Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.523908103Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.523920894Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.523939193Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685} {"msg":"Crucible 2d00d231-d978-4894-bf41-78959a4c38ff has session id: 7e366d1c-9b23-4fbb-8da8-3ab845ce50ed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524266167Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Upstairs opts: Upstairs UUID: 2d00d231-d978-4894-bf41-78959a4c38ff, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524304743Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Crucible stats registered with UUID: 2d00d231-d978-4894-bf41-78959a4c38ff","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524319994Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:22.524333245Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524504158Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"io task","client":"2","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524541684Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"io task","client":"0","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524557764Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"io task","client":"1","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524657721Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"task":"crutest"} {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524673142Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685} {"msg":"2d00d231-d978-4894-bf41-78959a4c38ff active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524704622Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524718593Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"client":"0","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524732574Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"client":"1","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524746506Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"client":"2","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524760247Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"io task","client":"2","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524774758Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"io task","client":"0","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524789049Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"io task","client":"1","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"ds_connection connected from Ok(127.0.0.1:56354)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524815003Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"io task","client":"1","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"ds_connection connected from Ok(127.0.0.1:33978)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524830133Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"io task","client":"2","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"ds_connection connected from Ok(127.0.0.1:44544)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.524919837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"io task","client":"0","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.525550261Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"client":"2","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.525578303Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"client":"1","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.525597681Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"client":"0","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.525899681Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.525943493Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"[0]R flush_numbers[0..12]: [216, 209, 209, 218, 218, 205, 213, 208, 216, 208, 211, 218]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526019406Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"[0]R generation[0..12]: [118, 116, 116, 119, 119, 113, 117, 115, 118, 115, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526036435Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526053045Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"[1]R flush_numbers[0..12]: [216, 209, 209, 218, 218, 205, 213, 208, 216, 208, 211, 218]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526069144Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"[1]R generation[0..12]: [118, 116, 116, 119, 119, 113, 117, 115, 118, 115, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526085264Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526101654Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"[2]R flush_numbers[0..12]: [216, 209, 209, 218, 218, 205, 213, 208, 216, 208, 211, 218]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526117394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"[2]R generation[0..12]: [118, 116, 116, 119, 119, 113, 117, 115, 118, 115, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526132764Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526148814Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Max found gen is 120","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:05:22.526162546Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526181484Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Next flush: 219","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526196395Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526209926Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"":"downstairs","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526224057Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526236739Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"client":"0","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.52625052Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"client":"1","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526264182Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"client":"2","":"downstairs","session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50edDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526282081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526295322Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"2d00d231-d978-4894-bf41-78959a4c38ff is now active with session: 7e366d1c-9b23-4fbb-8da8-3ab845ce50ed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526310233Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526322995Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685,"session_id":"7e366d1c-9b23-4fbb-8da8-3ab845ce50ed"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526336327Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685} {"msg":"Activated sub_volume 2d00d231-d978-4894-bf41-78959a4c38ff","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:22.526347649Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 291 len 32768 data: 13 14 11 10 9 9 8 8 02/30 Write block 35 len 16384 data: 10 11 10 8 03/30 Read block 154 len 32768 04/30 Read block 106 len 12288 05/30 Write block 66 len 36864 data: 11 10 12 13 14 14 11 10 14 06/30 Read block 309 len 4096 07/30 Read block 308 len 32768 08/30 Write block 114 len 40960 data: 6 5 5 6 5 5 5 3 4 6 09/30 Write block 248 len 16384 data: 21 20 21 18 10/30 Write block 47 len 36864 data: 5 5 6 6 6 9 10 10 9 11/30 Write block 141 len 20480 data: 8 7 8 9 6 12/30 Read block 236 len 12288 13/30 Flush 14/30 Flush 15/30 Write block 436 len 4096 data: 12 16/30 Write block 132 len 20480 data: 7 8 11 12 11 17/30 Write block 197 len 8192 data: 10 7 18/30 Read block 378 len 28672 19/30 Read block 356 len 32768 20/30 Flush 21/30 Write block 39 len 28672 data: 8 7 7 7 6 6 8 22/30 Read block 451 len 24576 23/30 Read block 548 len 12288 24/30 Write block 547 len 32768 data: 10 11 10 11 9 8 7 8 25/30 Read block 586 len 20480 26/30 Read block 501 len 16384 27/30 Read block 235 len 32768 28/30 Read block 88 len 12288 29/30 Read block 485 len 40960 30/30 Write block 365 len 16384 data: 8 11 14 13 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Read 1 Done Done Done false 1006 Acked Read 8 Done Done Done false 1007 Acked Write 10 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Write 9 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Read 3 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 1 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 23 1 25 Done 29 7 29 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:22.538888718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1685} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 216 216 216 F F F 1 020-039 120 116 120 219 209 219 T T T <--- 2 040-059 120 116 120 219 209 219 T F T <--- 3 060-079 120 119 120 219 218 219 F T F <--- 4 080-099 119 119 119 218 218 218 F F F 5 100-119 120 113 120 219 205 219 F F F <--- 6 120-139 120 117 120 221 213 221 F F F <--- 7 140-159 120 115 120 219 208 219 F F F <--- 8 160-179 118 118 118 216 216 216 F F F 9 180-199 120 115 120 221 208 221 F F F <--- 10 200-219 116 116 116 211 211 211 F F F 11 220-239 119 119 119 218 218 218 F F F 12 240-259 120 119 120 219 218 219 F F F <--- 13 260-279 117 117 117 213 213 213 F F F 14 280-299 120 116 120 219 209 219 F T F <--- 15 300-319 116 116 116 209 209 209 F F F 16 320-339 116 116 116 211 211 211 F F F 17 340-359 116 116 116 210 210 210 F F F 18 360-379 116 116 116 212 212 212 T F T <--- 19 380-399 116 116 116 212 212 212 F F F 20 400-419 118 118 118 217 217 217 F F F 21 420-439 120 111 120 221 202 221 F F F <--- 22 440-459 118 118 118 214 214 214 F F F 23 460-479 116 116 116 209 209 209 F F F 24 480-499 116 116 116 209 209 209 F F F 25 500-519 114 114 114 207 207 207 F F F 26 520-539 112 112 112 204 204 204 F F F 27 540-559 116 116 116 211 211 211 T F T <--- 28 560-579 114 114 114 206 206 206 F F F 29 580-599 118 118 118 216 216 216 F F F Max gen: 120, Max flush: 221 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 216 216 216 F F F 1 020-039 121 121 121 222 222 222 F F F 2 040-059 121 121 121 222 222 222 F F F 3 060-079 120 120 120 219 219 219 F F F 4 080-099 119 119 119 218 218 218 F F F 5 100-119 120 120 120 219 219 219 F F F 6 120-139 120 120 120 221 221 221 F F F 7 140-159 120 120 120 219 219 219 F F F 8 160-179 118 118 118 216 216 216 F F F 9 180-199 120 120 120 221 221 221 F F F 10 200-219 116 116 116 211 211 211 F F F 11 220-239 119 119 119 218 218 218 F F F 12 240-259 120 120 120 219 219 219 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 120 120 120 219 219 219 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 116 116 116 211 211 211 F F F 17 340-359 116 116 116 210 210 210 F F F 18 360-379 121 121 121 222 222 222 F F F 19 380-399 116 116 116 212 212 212 F F F 20 400-419 118 118 118 217 217 217 F F F 21 420-439 120 120 120 221 221 221 F F F 22 440-459 118 118 118 214 214 214 F F F 23 460-479 116 116 116 209 209 209 F F F 24 480-499 116 116 116 209 209 209 F F F 25 500-519 114 114 114 207 207 207 F F F 26 520-539 112 112 112 204 204 204 F F F 27 540-559 121 121 121 222 222 222 F F F 28 560-579 114 114 114 206 206 206 F F F 29 580-599 118 118 118 216 216 216 F F F Max gen: 121, Max flush: 222 Begin loop 61 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.857313699Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.868932478Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.869257074Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.869293111Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.869308421Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.86932685Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693} {"msg":"Crucible 4aa6f65e-708b-40ed-93fd-aeedba6523c9 has session id: 5ec6d51a-e052-4b73-8448-bb3bee092d46","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.869699426Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Upstairs opts: Upstairs UUID: 4aa6f65e-708b-40ed-93fd-aeedba6523c9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.869734014Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Crucible stats registered with UUID: 4aa6f65e-708b-40ed-93fd-aeedba6523c9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.869750024Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:26.869763465Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.869913981Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"io task","client":"0","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.869937606Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"io task","client":"1","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.869952397Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"io task","client":"2","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870094557Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"task":"crutest"} {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870109608Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693} {"msg":"4aa6f65e-708b-40ed-93fd-aeedba6523c9 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870127427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870140079Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"client":"0","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.87015403Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"client":"1","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870168191Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"client":"2","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870184531Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"io task","client":"2","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870198802Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"io task","client":"0","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870246422Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"io task","client":"1","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"ds_connection connected from Ok(127.0.0.1:34794)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870263801Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"io task","client":"2","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"ds_connection connected from Ok(127.0.0.1:61083)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870279421Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"io task","client":"0","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"ds_connection connected from Ok(127.0.0.1:33291)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870399476Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"io task","client":"1","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870903939Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"client":"0","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.870960273Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"client":"2","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871083966Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"client":"1","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871249891Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871288167Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"[0]R flush_numbers[0..12]: [216, 222, 222, 219, 218, 219, 221, 219, 216, 221, 211, 218]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871425901Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"[0]R generation[0..12]: [118, 121, 121, 120, 119, 120, 120, 120, 118, 120, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.87144296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.87145961Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"[1]R flush_numbers[0..12]: [216, 222, 222, 219, 218, 219, 221, 219, 216, 221, 211, 218]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.87147568Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"[1]R generation[0..12]: [118, 121, 121, 120, 119, 120, 120, 120, 118, 120, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871493998Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871510578Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"[2]R flush_numbers[0..12]: [216, 222, 222, 219, 218, 219, 221, 219, 216, 221, 211, 218]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871526178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"[2]R generation[0..12]: [118, 121, 121, 120, 119, 120, 120, 120, 118, 120, 116, 119]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871542468Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871558767Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:05:26.871572529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871590367Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Next flush: 223","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871603859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871617221Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"":"downstairs","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871631182Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871644443Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"client":"0","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871658395Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"client":"1","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:05:26.871672936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"client":"2","":"downstairs","session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871687886Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871703316Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"4aa6f65e-708b-40ed-93fd-aeedba6523c9 is now active with session: 5ec6d51a-e052-4b73-8448-bb3bee092d46","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871718287Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871731888Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693,"session_id":"5ec6d51a-e052-4b73-8448-bb3bee092d46"} {"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.87174585Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693} {"msg":"Activated sub_volume 4aa6f65e-708b-40ed-93fd-aeedba6523c9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:26.871757303Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 489 len 28672 data: 6 8 10 11 13 12 13 02/30 Flush 03/30 Write block 499 len 40960 data: 10 7 8 7 9 8 10 12 11 11 04/30 Read block 425 len 28672 05/30 Write block 115 len 32768 data: 6 6 7 6 6 6 4 5 06/30 Read block 277 len 4096 07/30 Read block 510 len 40960 08/30 Read block 221 len 16384 09/30 Read block 544 len 16384 10/30 Read block 248 len 4096 11/30 Write block 530 len 16384 data: 11 12 11 11 12/30 Write block 28 len 32768 data: 9 9 10 10 9 11 9 11 13/30 Write block 564 len 28672 data: 4 4 5 6 7 8 6 14/30 Read block 43 len 28672 15/30 Read block 397 len 36864 16/30 Read block 312 len 4096 17/30 Write block 523 len 40960 data: 9 9 11 12 13 12 12 12 13 12 18/30 Flush 19/30 Write block 0 len 40960 data: 3 4 6 9 8 8 7 6 5 6 20/30 Read block 247 len 28672 21/30 Flush 22/30 Write block 208 len 16384 data: 14 13 11 10 23/30 Read block 313 len 28672 24/30 Read block 553 len 32768 25/30 Read block 23 len 24576 26/30 Write block 216 len 36864 data: 14 15 10 10 9 7 7 9 10 27/30 Write block 581 len 28672 data: 13 14 16 16 11 10 9 28/30 Read block 222 len 36864 29/30 Write block 483 len 28672 data: 11 8 7 4 8 8 7 30/30 Write block 128 len 24576 data: 4 3 6 7 8 9 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Sent Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 10 Done Sent Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Read 1 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Read 4 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Read 1 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 10 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Read 7 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 2 34 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:26.885237271Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1693} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 122 118 122 225 216 225 F F F <--- 1 020-039 122 121 122 224 222 224 F F F <--- 2 040-059 121 121 121 222 222 222 F F F 3 060-079 120 120 120 219 219 219 F F F 4 080-099 119 119 119 218 218 218 F F F 5 100-119 122 120 122 224 219 224 F F F <--- 6 120-139 122 120 122 224 221 224 T F T <--- 7 140-159 120 120 120 219 219 219 F F F 8 160-179 118 118 118 216 216 216 F F F 9 180-199 120 120 120 221 221 221 F F F 10 200-219 116 116 116 211 211 211 T F T <--- 11 220-239 119 119 119 218 218 218 T F T <--- 12 240-259 120 120 120 219 219 219 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 120 120 120 219 219 219 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 116 116 116 211 211 211 F F F 17 340-359 116 116 116 210 210 210 F F F 18 360-379 121 121 121 222 222 222 F F F 19 380-399 116 116 116 212 212 212 F F F 20 400-419 118 118 118 217 217 217 F F F 21 420-439 120 120 120 221 221 221 F F F 22 440-459 118 118 118 214 214 214 F F F 23 460-479 116 116 116 209 209 209 F F F 24 480-499 122 116 122 224 209 224 T F T <--- 25 500-519 122 114 122 224 207 224 F F F <--- 26 520-539 122 112 122 224 204 224 F F F <--- 27 540-559 121 121 121 222 222 222 F F F 28 560-579 122 114 122 224 206 224 F F F <--- 29 580-599 118 118 118 216 216 216 T F T <--- Max gen: 122, Max flush: 225 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 122 122 122 225 225 225 F F F 1 020-039 122 122 122 224 224 224 F F F 2 040-059 121 121 121 222 222 222 F F F 3 060-079 120 120 120 219 219 219 F F F 4 080-099 119 119 119 218 218 218 F F F 5 100-119 122 122 122 224 224 224 F F F 6 120-139 123 123 123 226 226 226 F F F 7 140-159 120 120 120 219 219 219 F F F 8 160-179 118 118 118 216 216 216 F F F 9 180-199 120 120 120 221 221 221 F F F 10 200-219 123 123 123 226 226 226 F F F 11 220-239 123 123 123 226 226 226 F F F 12 240-259 120 120 120 219 219 219 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 120 120 120 219 219 219 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 116 116 116 211 211 211 F F F 17 340-359 116 116 116 210 210 210 F F F 18 360-379 121 121 121 222 222 222 F F F 19 380-399 116 116 116 212 212 212 F F F 20 400-419 118 118 118 217 217 217 F F F 21 420-439 120 120 120 221 221 221 F F F 22 440-459 118 118 118 214 214 214 F F F 23 460-479 116 116 116 209 209 209 F F F 24 480-499 123 123 123 226 226 226 F F F 25 500-519 122 122 122 224 224 224 F F F 26 520-539 122 122 122 224 224 224 F F F 27 540-559 121 121 121 222 222 222 F F F 28 560-579 122 122 122 224 224 224 F F F 29 580-599 123 123 123 226 226 226 F F F Max gen: 123, Max flush: 226 Begin loop 62 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.202277592Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.213975731Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.21429484Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.214331967Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.214344189Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.214362028Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701} {"msg":"Crucible b8a0a084-2f6f-4689-80dc-77033e98e3e7 has session id: 9b8fc48d-bf40-46b8-a3be-a8b8785d7328","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.214709979Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Upstairs opts: Upstairs UUID: b8a0a084-2f6f-4689-80dc-77033e98e3e7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.214744697Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Crucible stats registered with UUID: b8a0a084-2f6f-4689-80dc-77033e98e3e7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.214760198Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:31.214775868Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.214939185Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"io task","client":"2","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.214973394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"io task","client":"1","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.214989234Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"io task","client":"0","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.215088661Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"task":"crutest"} {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.215104131Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701} {"msg":"b8a0a084-2f6f-4689-80dc-77033e98e3e7 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.21512188Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.215134942Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"client":"0","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.215148873Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"client":"1","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.215162905Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"client":"2","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.215176326Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"io task","client":"0","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.215190567Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"io task","client":"2","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.215204958Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"io task","client":"1","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"ds_connection connected from Ok(127.0.0.1:40985)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.215270487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"io task","client":"1","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"ds_connection connected from Ok(127.0.0.1:44900)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.21529811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"io task","client":"0","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"ds_connection connected from Ok(127.0.0.1:58578)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.215314199Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"io task","client":"2","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216075491Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"client":"0","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216107301Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"client":"1","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216164615Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"client":"2","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216438383Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216483894Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"[0]R flush_numbers[0..12]: [225, 224, 222, 219, 218, 224, 226, 219, 216, 221, 226, 226]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216539669Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"[0]R generation[0..12]: [122, 122, 121, 120, 119, 122, 123, 120, 118, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216556859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216573738Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"[1]R flush_numbers[0..12]: [225, 224, 222, 219, 218, 224, 226, 219, 216, 221, 226, 226]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216589488Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"[1]R generation[0..12]: [122, 122, 121, 120, 119, 122, 123, 120, 118, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216605228Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216621368Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"[2]R flush_numbers[0..12]: [225, 224, 222, 219, 218, 224, 226, 219, 216, 221, 226, 226]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216637098Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"[2]R generation[0..12]: [122, 122, 121, 120, 119, 122, 123, 120, 118, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216652668Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216669108Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216684898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairsWait for a query_work_queue command to finish before sending IO ","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216704965Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Next flush: 227","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216718867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216732398Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"":"downstairs","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216746949Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216759571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"client":"0","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216773882Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"client":"1","":"downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216787913Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"client":"2","":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 downstairs","session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216805242Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216818624Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"b8a0a084-2f6f-4689-80dc-77033e98e3e7 is now active with session: 9b8fc48d-bf40-46b8-a3be-a8b8785d7328","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216833505Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216846397Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701,"session_id":"9b8fc48d-bf40-46b8-a3be-a8b8785d7328"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216859668Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701} {"msg":"Activated sub_volume b8a0a084-2f6f-4689-80dc-77033e98e3e7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:31.216871131Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 388 len 36864 data: 11 12 12 9 8 9 9 8 9 02/30 Write block 138 len 24576 data: 9 9 8 9 8 9 03/30 Write block 418 len 32768 data: 10 11 10 12 10 11 9 7 04/30 Write block 367 len 16384 data: 15 14 11 10 05/30 Flush 06/30 Write block 164 len 16384 data: 9 10 10 10 07/30 Read block 461 len 4096 08/30 Read block 123 len 4096 09/30 Flush 10/30 Write block 459 len 20480 data: 13 10 8 6 5 11/30 Write block 485 len 16384 data: 8 5 9 9 12/30 Write block 498 len 36864 data: 9 11 8 9 8 10 9 11 13 13/30 Flush 14/30 Write block 117 len 32768 data: 8 7 7 7 5 6 7 9 15/30 Write block 446 len 8192 data: 13 12 16/30 Read block 473 len 40960 17/30 Read block 45 len 4096 18/30 Write block 35 len 16384 data: 12 12 11 9 19/30 Read block 346 len 36864 20/30 Read block 294 len 24576 21/30 Read block 142 len 32768 22/30 Read block 164 len 24576 23/30 Read block 356 len 12288 24/30 Write block 435 len 36864 data: 12 13 10 9 9 10 11 14 15 25/30 Write block 373 len 36864 data: 8 8 10 10 8 10 11 12 10 26/30 Flush 27/30 Write block 282 len 40960 data: 7 7 10 10 7 8 9 8 9 14 28/30 Write block 377 len 24576 data: 9 11 12 13 11 8 29/30 Write block 162 len 4096 data: 9 30/30 Read block 225 len 32768 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 6 Done Sent Done false 1002 Acked Write 8 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 4 Done Sent Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 5 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 2 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Write 4 Done Sent Done false 1018 Acked Read 9 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Write 1 Done Sent Done false 1029 Acked Read 8 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 0 30 Done 29 1 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:31.228801174Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1701} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 122 122 122 225 225 225 F F F 1 020-039 124 122 124 230 224 230 F F F <--- 2 040-059 121 121 121 222 222 222 F F F 3 060-079 120 120 120 219 219 219 F F F 4 080-099 119 119 119 218 218 218 F F F 5 100-119 124 122 124 230 224 230 F F F <--- 6 120-139 124 123 124 230 226 230 F F F <--- 7 140-159 124 120 124 227 219 227 F F F <--- 8 160-179 124 118 124 228 216 228 T F T <--- 9 180-199 120 120 120 221 221 221 F F F 10 200-219 123 123 123 226 226 226 F F F 11 220-239 123 123 123 226 226 226 F F F 12 240-259 120 120 120 219 219 219 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 120 120 120 219 219 219 T F T <--- 15 300-319 116 116 116 209 209 209 F F F 16 320-339 116 116 116 211 211 211 F F F 17 340-359 116 116 116 210 210 210 F F F 18 360-379 124 121 124 230 222 230 T F T <--- 19 380-399 124 116 124 230 212 230 T T T <--- 20 400-419 124 118 124 227 217 227 F F F <--- 21 420-439 124 120 124 230 221 230 F F F <--- 22 440-459 124 118 124 230 214 230 F F F <--- 23 460-479 124 116 124 229 209 229 F F F <--- 24 480-499 124 123 124 229 226 229 F F F <--- 25 500-519 124 122 124 229 224 229 F F F <--- 26 520-539 122 122 122 224 224 224 F F F 27 540-559 121 121 121 222 222 222 F F F 28 560-579 122 122 122 224 224 224 F F F 29 580-599 123 123 123 226 226 226 F F F Max gen: 124, Max flush: 230 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 122 122 122 225 225 225 F F F 1 020-039 124 124 124 230 230 230 F F F 2 040-059 121 121 121 222 222 222 F F F 3 060-079 120 120 120 219 219 219 F F F 4 080-099 119 119 119 218 218 218 F F F 5 100-119 124 124 124 230 230 230 F F F 6 120-139 124 124 124 230 230 230 F F F 7 140-159 124 124 124 227 227 227 F F F 8 160-179 125 125 125 231 231 231 F F F 9 180-199 120 120 120 221 221 221 F F F 10 200-219 123 123 123 226 226 226 F F F 11 220-239 123 123 123 226 226 226 F F F 12 240-259 120 120 120 219 219 219 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 125 125 125 231 231 231 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 116 116 116 211 211 211 F F F 17 340-359 116 116 116 210 210 210 F F F 18 360-379 125 125 125 231 231 231 F F F 19 380-399 125 125 125 231 231 231 F F F 20 400-419 124 124 124 227 227 227 F F F 21 420-439 124 124 124 230 230 230 F F F 22 440-459 124 124 124 230 230 230 F F F 23 460-479 124 124 124 229 229 229 F F F 24 480-499 124 124 124 229 229 229 F F F 25 500-519 124 124 124 229 229 229 F F F 26 520-539 122 122 122 224 224 224 F F F 27 540-559 121 121 121 222 222 222 F F F 28 560-579 122 122 122 224 224 224 F F F 29 580-599 123 123 123 226 226 226 F F F Max gen: 125, Max flush: 231 Begin loop 63 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.550073675Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.56174566Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562073125Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562115358Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.56212859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562146648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709} {"msg":"Crucible 396e0e14-fc23-4c46-9d2f-b93697b01a64 has session id: ec2b7d98-a787-4849-b68e-27d17f4732ed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562470635Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Upstairs opts: Upstairs UUID: 396e0e14-fc23-4c46-9d2f-b93697b01a64, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.5625102Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Crucible stats registered with UUID: 396e0e14-fc23-4c46-9d2f-b93697b01a64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.56252599Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:35.562539292Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562695364Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"io task","client":"0","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562719828Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"io task","client":"1","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562734729Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"io task","client":"2","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.56286055Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"task":"crutest"} {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.56287599Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709} {"msg":"396e0e14-fc23-4c46-9d2f-b93697b01a64 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562893409Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562906201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"client":"0","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562922751Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"client":"1","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562937262Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"client":"2","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562950893Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"io task","client":"2","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562965244Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"io task","client":"0","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.562979585Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"io task","client":"1","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"ds_connection connected from Ok(127.0.0.1:34924)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.563090335Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"io task","client":"0","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"ds_connection connected from Ok(127.0.0.1:39626)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.563174313Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"io task","client":"2","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"ds_connection connected from Ok(127.0.0.1:44338)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.563202525Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"io task","client":"1","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.563711135Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"client":"2","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.563794273Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"client":"0","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.563848469Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"client":"1","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564081103Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564122177Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"[0]R flush_numbers[0..12]: [225, 230, 222, 219, 218, 230, 230, 227, 231, 221, 226, 226]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564183918Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"[0]R generation[0..12]: [122, 124, 121, 120, 119, 124, 124, 124, 125, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564202526Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564222244Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"[1]R flush_numbers[0..12]: [225, 230, 222, 219, 218, 230, 230, 227, 231, 221, 226, 226]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564238704Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"[1]R generation[0..12]: [122, 124, 121, 120, 119, 124, 124, 124, 125, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564254524Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564270714Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"[2]R flush_numbers[0..12]: [225, 230, 222, 219, 218, 230, 230, 227, 231, 221, 226, 226]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564286374Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"[2]R generation[0..12]: [122, 124, 121, 120, 119, 124, 124, 124, 125, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564302394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564318594Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:05:35.564332465Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564350084Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Next flush: 232","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564363595Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564377077Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"":"downstairs","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564391668Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.5644044Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"client":"0","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564418751Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"client":"1","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564437379Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"client":"2","":"downstairs","session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564452669Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564466111Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"396e0e14-fc23-4c46-9d2f-b93697b01a64 is now active with session: ec2b7d98-a787-4849-b68e-27d17f4732ed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564481291Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564494283Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709,"session_id":"ec2b7d98-a787-4849-b68e-27d17f4732ed"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564507515Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709} {"msg":"Activated sub_volume 396e0e14-fc23-4c46-9d2f-b93697b01a64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:35.564518968Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 154 len 24576 data: 7 10 11 13 15 13 03/30 Write block 564 len 16384 data: 5 5 6 7 04/30 Write block 97 len 16384 data: 5 3 5 5 05/30 Read block 196 len 12288 06/30 Read block 364 len 36864 07/30 Write block 362 len 4096 data: 11 08/30 Read block 464 len 40960 09/30 Read block 156 len 4096 10/30 Read block 336 len 32768 11/30 Read block 467 len 8192 12/30 Write block 160 len 4096 data: 9 13/30 Read block 413 len 40960 14/30 Read block 511 len 36864 15/30 Read block 575 len 12288 16/30 Write block 368 len 16384 data: 15 12 11 10 17/30 Flush 18/30 Write block 392 len 32768 data: 9 10 10 9 10 9 8 9 19/30 Write block 211 len 12288 data: 11 12 16 20/30 Read block 564 len 40960 21/30 Read block 298 len 8192 22/30 Read block 269 len 16384 23/30 Write block 230 len 40960 data: 5 5 8 10 11 14 15 11 14 14 24/30 Read block 88 len 8192 25/30 Read block 227 len 36864 26/30 Read block 232 len 12288 27/30 Read block 331 len 4096 28/30 Write block 285 len 8192 data: 11 8 29/30 Write block 326 len 12288 data: 10 10 11 30/30 Write block 146 len 36864 data: 8 7 8 7 8 7 9 7 8 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 6 Done Sent Done false 1002 Acked Write 4 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Read 9 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Read 1 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Read 2 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Write 4 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Read 4 Done Sent Done false 1022 Acked Write 10 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Write 3 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1000)) Some(JobId(1016)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:35.575659207Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1709} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 122 122 122 225 225 225 F F F 1 020-039 124 124 124 230 230 230 F F F 2 040-059 121 121 121 222 222 222 F F F 3 060-079 120 120 120 219 219 219 F F F 4 080-099 126 119 126 233 218 233 F F F <--- 5 100-119 126 124 126 233 230 233 F F F <--- 6 120-139 124 124 124 230 230 230 F F F 7 140-159 126 124 126 233 227 233 T F T <--- 8 160-179 126 125 126 233 231 233 F F F <--- 9 180-199 120 120 120 221 221 221 F F F 10 200-219 123 123 123 226 226 226 T F T <--- 11 220-239 123 123 123 226 226 226 T F T <--- 12 240-259 120 120 120 219 219 219 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 125 125 125 231 231 231 T F T <--- 15 300-319 116 116 116 209 209 209 F F F 16 320-339 116 116 116 211 211 211 T F T <--- 17 340-359 116 116 116 210 210 210 F F F 18 360-379 126 125 126 233 231 233 F F F <--- 19 380-399 125 125 125 231 231 231 T F T <--- 20 400-419 124 124 124 227 227 227 F F F 21 420-439 124 124 124 230 230 230 F F F 22 440-459 124 124 124 230 230 230 F F F 23 460-479 124 124 124 229 229 229 F F F 24 480-499 124 124 124 229 229 229 F F F 25 500-519 124 124 124 229 229 229 F F F 26 520-539 122 122 122 224 224 224 F F F 27 540-559 121 121 121 222 222 222 F F F 28 560-579 126 122 126 233 224 233 F F F <--- 29 580-599 123 123 123 226 226 226 F F F Max gen: 126, Max flush: 233 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 122 122 122 225 225 225 F F F 1 020-039 124 124 124 230 230 230 F F F 2 040-059 121 121 121 222 222 222 F F F 3 060-079 120 120 120 219 219 219 F F F 4 080-099 126 126 126 233 233 233 F F F 5 100-119 126 126 126 233 233 233 F F F 6 120-139 124 124 124 230 230 230 F F F 7 140-159 127 127 127 234 234 234 F F F 8 160-179 126 126 126 233 233 233 F F F 9 180-199 120 120 120 221 221 221 F F F 10 200-219 127 127 127 234 234 234 F F F 11 220-239 127 127 127 234 234 234 F F F 12 240-259 120 120 120 219 219 219 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 127 127 127 234 234 234 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 127 127 127 234 234 234 F F F 17 340-359 116 116 116 210 210 210 F F F 18 360-379 126 126 126 233 233 233 F F F 19 380-399 127 127 127 234 234 234 F F F 20 400-419 124 124 124 227 227 227 F F F 21 420-439 124 124 124 230 230 230 F F F 22 440-459 124 124 124 230 230 230 F F F 23 460-479 124 124 124 229 229 229 F F F 24 480-499 124 124 124 229 229 229 F F F 25 500-519 124 124 124 229 229 229 F F F 26 520-539 122 122 122 224 224 224 F F F 27 540-559 121 121 121 222 222 222 F F F 28 560-579 126 126 126 233 233 233 F F F 29 580-599 123 123 123 226 226 226 F F F Max gen: 127, Max flush: 234 Begin loop 64 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.894961506Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:39.90555313Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.905605957Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.915994179Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.916287705Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.91632658Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.916338842Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.916356681Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717} {"msg":"Crucible dea3e7fd-6236-431e-a342-b10ed9409020 has session id: 43b5b87a-ec72-4643-bf2d-9f724d58673f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.916714237Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Upstairs opts: Upstairs UUID: dea3e7fd-6236-431e-a342-b10ed9409020, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.916757529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Crucible stats registered with UUID: dea3e7fd-6236-431e-a342-b10ed9409020","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.916773869Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:39.916787201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.916928312Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"io task","client":"2","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.916952137Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"io task","client":"0","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.916967138Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"io task","client":"1","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.91709102Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"task":"crutest"} {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.91710623Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717} {"msg":"dea3e7fd-6236-431e-a342-b10ed9409020 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.917127317Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.917140319Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"client":"0","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.91715459Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"client":"1","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.917168671Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"client":"2","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.917213313Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"io task","client":"2","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.917238497Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"io task","client":"0","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.917253847Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"io task","client":"1","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"ds_connection connected from Ok(127.0.0.1:45647)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.917268898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"io task","client":"1","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"ds_connection connected from Ok(127.0.0.1:43073)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.917286537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"io task","client":"2","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"ds_connection connected from Ok(127.0.0.1:48648)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.917301857Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"io task","client":"0","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918018777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"client":"1","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918046549Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"client":"2","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.91810934Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"client":"0","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918360732Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918399438Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"[0]R flush_numbers[0..12]: [225, 230, 222, 219, 233, 233, 230, 234, 233, 221, 234, 234]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.91847425Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"[0]R generation[0..12]: [122, 124, 121, 120, 126, 126, 124, 127, 126, 120, 127, 127]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918492979Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918509778Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"[1]R flush_numbers[0..12]: [225, 230, 222, 219, 233, 233, 230, 234, 233, 221, 234, 234]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918525768Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"[1]R generation[0..12]: [122, 124, 121, 120, 126, 126, 124, 127, 126, 120, 127, 127]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918541668Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918558238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"[2]R flush_numbers[0..12]: [225, 230, 222, 219, 233, 233, 230, 234, 233, 221, 234, 234]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918573918Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"[2]R generation[0..12]: [122, 124, 121, 120, 126, 126, 124, 127, 126, 120, 127, 127]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918591957Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918608736Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Max found gen is 128","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2026-03-14T05:05:39.918627534Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918641765Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Next flush: 235","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918655137Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918668588Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"":"downstairs","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918683269Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918696061Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"client":"0","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918710312Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"client":"1","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918724343Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :1717,"client":"2","":"downstairs","session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918741503Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918754954Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"dea3e7fd-6236-431e-a342-b10ed9409020 is now active with session: 43b5b87a-ec72-4643-bf2d-9f724d58673f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918769845Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918782427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717,"session_id":"43b5b87a-ec72-4643-bf2d-9f724d58673f"} {"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.918796038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717} {"msg":"Activated sub_volume dea3e7fd-6236-431e-a342-b10ed9409020","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:39.91880906Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 588 len 20480 data: 8 10 9 7 10 03/30 Write block 425 len 36864 data: 8 6 7 5 7 5 7 10 10 04/30 Flush 05/30 Write block 133 len 32768 data: 10 12 13 12 11 10 10 9 06/30 Read block 271 len 20480 07/30 Read block 236 len 24576 08/30 Flush 09/30 Write block 425 len 20480 data: 9 7 8 6 8 10/30 Read block 410 len 16384 11/30 Read block 136 len 16384 12/30 Read block 343 len 32768 13/30 Write block 343 len 28672 data: 9 8 9 9 10 11 12 14/30 Write block 9 len 4096 data: 7 15/30 Write block 354 len 40960 data: 8 8 10 9 8 7 10 11 12 9 16/30 Flush 17/30 Write block 14 len 8192 data: 10 11 18/30 Flush 19/30 Write block 537 len 16384 data: 6 7 7 7 20/30 Read block 5 len 20480 21/30 Flush 22/30 Write block 482 len 32768 data: 11 12 9 9 6 10 10 8 23/30 Write block 354 len 8192 data: 9 9 24/30 Write block 65 len 24576 data: 10 12 11 13 14 15 25/30 Read block 0 len 12288 26/30 Write block 29 len 36864 data: 10 11 11 10 12 10 13 13 12 27/30 Read block 344 len 36864 28/30 Read block 166 len 4096 29/30 Write block 94 len 32768 data: 5 7 7 6 4 6 6 8 30/30 Write block 27 len 16384 data: 7 10 11 12 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 5 Done Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 8 Sent Done Done false 1005 Acked Read 5 Sent Done Done false 1006 Acked Read 6 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 5 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Write 10 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 2 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 4 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Write 6 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 2 2 32 Done 1 27 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:39.931181995Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1717} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 122 128 128 225 239 239 F F F <--- 1 020-039 124 124 124 230 230 230 F T T <--- 2 040-059 121 121 121 222 222 222 F F F 3 060-079 120 120 120 219 219 219 F T T <--- 4 080-099 126 126 126 233 233 233 F T T <--- 5 100-119 126 126 126 233 233 233 F T T <--- 6 120-139 124 128 128 230 237 237 F F F <--- 7 140-159 127 128 128 234 237 237 F F F <--- 8 160-179 126 126 126 233 233 233 F F F 9 180-199 120 120 120 221 221 221 F F F 10 200-219 127 127 127 234 234 234 F F F 11 220-239 127 127 127 234 234 234 F F F 12 240-259 120 120 120 219 219 219 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 127 127 127 234 234 234 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 127 127 127 234 234 234 F F F 17 340-359 116 128 128 210 238 238 F T T <--- 18 360-379 126 128 128 233 238 238 F F F <--- 19 380-399 127 127 127 234 234 234 F F F 20 400-419 124 124 124 227 227 227 F F F 21 420-439 124 128 128 230 238 238 F F F <--- 22 440-459 124 124 124 230 230 230 F F F 23 460-479 124 124 124 229 229 229 F F F 24 480-499 124 124 124 229 229 229 F T T <--- 25 500-519 124 124 124 229 229 229 F F F 26 520-539 122 128 128 224 240 240 F F F <--- 27 540-559 121 128 128 222 240 240 F F F <--- 28 560-579 126 126 126 233 233 233 F F F 29 580-599 123 128 128 226 236 236 T F F <--- Max gen: 128, Max flush: 240 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 239 239 239 F F F 1 020-039 129 129 129 241 241 241 F F F 2 040-059 121 121 121 222 222 222 F F F 3 060-079 129 129 129 241 241 241 F F F 4 080-099 129 129 129 241 241 241 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 128 128 128 237 237 237 F F F 7 140-159 128 128 128 237 237 237 F F F 8 160-179 126 126 126 233 233 233 F F F 9 180-199 120 120 120 221 221 221 F F F 10 200-219 127 127 127 234 234 234 F F F 11 220-239 127 127 127 234 234 234 F F F 12 240-259 120 120 120 219 219 219 F F F 13 260-279 117 117 117 213 213 213 F F F 14 280-299 127 127 127 234 234 234 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 127 127 127 234 234 234 F F F 17 340-359 129 129 129 241 241 241 F F F 18 360-379 128 128 128 238 238 238 F F F 19 380-399 127 127 127 234 234 234 F F F 20 400-419 124 124 124 227 227 227 F F F 21 420-439 128 128 128 238 238 238 F F F 22 440-459 124 124 124 230 230 230 F F F 23 460-479 124 124 124 229 229 229 F F F 24 480-499 129 129 129 241 241 241 F F F 25 500-519 124 124 124 229 229 229 F F F 26 520-539 128 128 128 240 240 240 F F F 27 540-559 128 128 128 240 240 240 F F F 28 560-579 126 126 126 233 233 233 F F F 29 580-599 128 128 128 236 236 236 F F F Max gen: 129, Max flush: 241 Begin loop 65 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.264430231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:44.275083946Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.27514047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.285367353Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.285662458Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.285702113Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.285714245Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.285731814Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725} {"msg":"Crucible ca3ccc4e-6aa6-4388-a9ae-38ea1b8fccd2 has session id: 1bd37a5d-97c8-4a23-95bf-87ff5772e2a3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286063336Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Upstairs opts: Upstairs UUID: ca3ccc4e-6aa6-4388-a9ae-38ea1b8fccd2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286097454Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Crucible stats registered with UUID: ca3ccc4e-6aa6-4388-a9ae-38ea1b8fccd2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286112255Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:44.286125187Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286307093Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"io task","client":"1","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286330438Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"io task","client":"0","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286347777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"io task","client":"2","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.2864871Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"task":"crutest"} {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.28650228Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725} {"msg":"ca3ccc4e-6aa6-4388-a9ae-38ea1b8fccd2 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286523267Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286536049Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"client":"0","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.28655034Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"client":"1","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286564181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"client":"2","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286577463Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"io task","client":"2","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286591634Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"io task","client":"0","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286605945Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"io task","client":"1","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"ds_connection connected from Ok(127.0.0.1:40564)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286763656Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"io task","client":"2","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"ds_connection connected from Ok(127.0.0.1:59958)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286861474Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"io task","client":"0","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"ds_connection connected from Ok(127.0.0.1:36441)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.286890076Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"io task","client":"1","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287412718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"client":"1","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287448555Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"client":"2","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287464885Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"client":"0","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287799735Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287844846Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"[0]R flush_numbers[0..12]: [239, 241, 222, 241, 241, 241, 237, 237, 233, 221, 234, 234]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287888939Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"[0]R generation[0..12]: [128, 129, 121, 129, 129, 129, 128, 128, 126, 120, 127, 127]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287906038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287922568Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"[1]R flush_numbers[0..12]: [239, 241, 222, 241, 241, 241, 237, 237, 233, 221, 234, 234]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287938528Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"[1]R generation[0..12]: [128, 129, 121, 129, 129, 129, 128, 128, 126, 120, 127, 127]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287954777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287971257Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"[2]R flush_numbers[0..12]: [239, 241, 222, 241, 241, 241, 237, 237, 233, 221, 234, 234]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.287986717Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"[2]R generation[0..12]: [128, 129, 121, 129, 129, 129, 128, 128, 126, 120, 127, 127]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288001978Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288018227Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288031679Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:05:44.28804593Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Next flush: 242","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288064169Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.28807802Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"":"downstairs","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288091941Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288104663Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"client":"0","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288118474Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"client":"1","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288132046Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"client":"2","":"downstairs","session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288145977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288160628Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"ca3ccc4e-6aa6-4388-a9ae-38ea1b8fccd2 is now active with session: 1bd37a5d-97c8-4a23-95bf-87ff5772e2a3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288176178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.28818883Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725,"session_id":"1bd37a5d-97c8-4a23-95bf-87ff5772e2a3"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288202142Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725} {"msg":"Activated sub_volume ca3ccc4e-6aa6-4388-a9ae-38ea1b8fccd2","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:44.288213825Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 223 len 8192 data: 10 11 02/30 Read block 397 len 8192 03/30 Flush 04/30 Write block 479 len 4096 data: 12 05/30 Read block 214 len 20480 06/30 Read block 581 len 40960 07/30 Read block 105 len 20480 08/30 Read block 28 len 8192 09/30 Write block 437 len 24576 data: 11 10 10 11 12 15 10/30 Read block 131 len 40960 11/30 Write block 387 len 40960 data: 10 12 13 13 10 10 11 11 10 11 12/30 Read block 526 len 32768 13/30 Read block 525 len 16384 14/30 Write block 246 len 20480 data: 14 17 22 21 22 15/30 Write block 221 len 40960 data: 8 8 11 12 8 7 9 9 5 6 16/30 Read block 577 len 28672 17/30 Read block 348 len 24576 18/30 Read block 67 len 28672 19/30 Write block 269 len 36864 data: 11 11 8 8 7 9 10 10 8 20/30 Read block 438 len 12288 21/30 Write block 357 len 24576 data: 10 9 8 11 12 13 22/30 Write block 567 len 24576 data: 8 8 9 7 7 8 23/30 Flush 24/30 Write block 341 len 40960 data: 9 11 10 9 10 10 11 12 13 13 25/30 Read block 227 len 4096 26/30 Read block 539 len 24576 27/30 Read block 60 len 12288 28/30 Write block 230 len 28672 data: 7 6 9 11 12 15 16 29/30 Read block 314 len 8192 30/30 Read block 157 len 32768 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 1 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Read 10 Sent Done Done false 1006 Acked Read 5 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Write 6 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Write 10 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Write 10 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Read 8 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 0 29 Done 2 29 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:44.301873191Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1725} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 239 239 239 F F F 1 020-039 129 129 129 241 241 241 F F F 2 040-059 121 121 121 222 222 222 F F F 3 060-079 129 129 129 241 241 241 F F F 4 080-099 129 129 129 241 241 241 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 128 128 128 237 237 237 F F F 7 140-159 128 128 128 237 237 237 F F F 8 160-179 126 126 126 233 233 233 F F F 9 180-199 120 120 120 221 221 221 F F F 10 200-219 127 127 127 234 234 234 F F F 11 220-239 127 130 130 234 243 243 T T T <--- 12 240-259 120 130 130 219 243 243 F F F <--- 13 260-279 117 130 130 213 243 243 F F F <--- 14 280-299 127 127 127 234 234 234 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 127 127 127 234 234 234 F F F 17 340-359 129 130 130 241 243 243 F T T <--- 18 360-379 128 130 130 238 243 243 F F F <--- 19 380-399 127 130 130 234 243 243 F F F <--- 20 400-419 124 124 124 227 227 227 F F F 21 420-439 128 130 130 238 243 243 F F F <--- 22 440-459 124 130 130 230 243 243 F F F <--- 23 460-479 124 130 130 229 243 243 F F F <--- 24 480-499 129 129 129 241 241 241 F F F 25 500-519 124 124 124 229 229 229 F F F 26 520-539 128 128 128 240 240 240 F F F 27 540-559 128 128 128 240 240 240 F F F 28 560-579 126 130 130 233 243 243 F F F <--- 29 580-599 128 128 128 236 236 236 F F F Max gen: 130, Max flush: 243 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 239 239 239 F F F 1 020-039 129 129 129 241 241 241 F F F 2 040-059 121 121 121 222 222 222 F F F 3 060-079 129 129 129 241 241 241 F F F 4 080-099 129 129 129 241 241 241 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 128 128 128 237 237 237 F F F 7 140-159 128 128 128 237 237 237 F F F 8 160-179 126 126 126 233 233 233 F F F 9 180-199 120 120 120 221 221 221 F F F 10 200-219 127 127 127 234 234 234 F F F 11 220-239 131 131 131 244 244 244 F F F 12 240-259 130 130 130 243 243 243 F F F 13 260-279 130 130 130 243 243 243 F F F 14 280-299 127 127 127 234 234 234 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 127 127 127 234 234 234 F F F 17 340-359 131 131 131 244 244 244 F F F 18 360-379 130 130 130 243 243 243 F F F 19 380-399 130 130 130 243 243 243 F F F 20 400-419 124 124 124 227 227 227 F F F 21 420-439 130 130 130 243 243 243 F F F 22 440-459 130 130 130 243 243 243 F F F 23 460-479 130 130 130 243 243 243 F F F 24 480-499 129 129 129 241 241 241 F F F 25 500-519 124 124 124 229 229 229 F F F 26 520-539 128 128 128 240 240 240 F F F 27 540-559 128 128 128 240 240 240 F F F 28 560-579 130 130 130 243 243 243 F F F 29 580-599 128 128 128 236 236 236 F F F Max gen: 131, Max flush: 244 Begin loop 66 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.623212081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.63497424Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.635298666Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.635334633Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.635349923Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.635367682Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733} {"msg":"Crucible f85be54e-eca6-4f16-ab43-6849063f339d has session id: ac604235-00a7-4964-8e80-24b24890b262","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.635727616Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Upstairs opts: Upstairs UUID: f85be54e-eca6-4f16-ab43-6849063f339d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.635761875Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Crucible stats registered with UUID: f85be54e-eca6-4f16-ab43-6849063f339d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.635777675Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:48.635790856Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.635948977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"io task","client":"1","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.635982716Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"io task","client":"0","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.635998556Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"io task","client":"2","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636113364Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"task":"crutest"} {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636128884Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733} {"msg":"f85be54e-eca6-4f16-ab43-6849063f339d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636145304Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636158066Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"client":"0","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636171997Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"client":"1","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636186957Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"client":"2","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636201019Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"io task","client":"2","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636217498Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"io task","client":"0","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636232029Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"io task","client":"1","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"ds_connection connected from Ok(127.0.0.1:36209)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.63624685Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"io task","client":"2","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"ds_connection connected from Ok(127.0.0.1:41270)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636300006Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"io task","client":"1","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"ds_connection connected from Ok(127.0.0.1:42521)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.63634132Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"io task","client":"0","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.636986375Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"client":"0","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637009421Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"client":"2","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.63702611Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"client":"1","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637370214Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637385624Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"[0]R flush_numbers[0..12]: [239, 241, 222, 241, 241, 241, 237, 237, 233, 221, 234, 244]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637458558Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"[0]R generation[0..12]: [128, 129, 121, 129, 129, 129, 128, 128, 126, 120, 127, 131]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637475887Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637492367Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"[1]R flush_numbers[0..12]: [239, 241, 222, 241, 241, 241, 237, 237, 233, 221, 234, 244]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637507977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"[1]R generation[0..12]: [128, 129, 121, 129, 129, 129, 128, 128, 126, 120, 127, 131]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637525536Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637541936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"[2]R flush_numbers[0..12]: [239, 241, 222, 241, 241, 241, 237, 237, 233, 221, 234, 244]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637557706Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"[2]R generation[0..12]: [128, 129, 121, 129, 129, 129, 128, 128, 126, 120, 127, 131]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637573396Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637589726Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637603277Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637623575Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Next flush: 245","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637637376Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637651097Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"":"downstairs","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637664989Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637677511Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"client":"0","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637691032Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"client":"1","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637704684Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"client":"2","":"downstairs","session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"All required reconciliation work is completed",Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637722073Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637735724Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"f85be54e-eca6-4f16-ab43-6849063f339d is now active with session: ac604235-00a7-4964-8e80-24b24890b262","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637752394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637765366Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733,"session_id":"ac604235-00a7-4964-8e80-24b24890b262"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.637778657Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733} {"msg":"Activated sub_volume f85be54e-eca6-4f16-ab43-6849063f339d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:48.63779021Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 593 len 28672 data: 10 6 5 5 4 4 3 02/30 Read block 521 len 8192 03/30 Write block 183 len 8192 data: 7 9 04/30 Flush 05/30 Write block 182 len 12288 data: 7 8 10 06/30 Write block 135 len 36864 data: 14 13 12 11 11 10 10 9 10 07/30 Write block 281 len 20480 data: 6 8 8 11 12 08/30 Read block 428 len 40960 09/30 Write block 21 len 8192 data: 11 11 10/30 Read block 28 len 32768 11/30 Read block 256 len 20480 12/30 Read block 128 len 20480 13/30 Write block 505 len 12288 data: 12 14 12 14/30 Write block 264 len 36864 data: 12 12 12 11 11 12 12 9 9 15/30 Read block 9 len 24576 16/30 Write block 507 len 24576 data: 13 12 11 10 9 10 17/30 Write block 190 len 20480 data: 8 9 8 7 9 18/30 Read block 125 len 24576 19/30 Read block 120 len 28672 20/30 Read block 492 len 4096 21/30 Read block 456 len 32768 22/30 Read block 179 len 40960 23/30 Write block 386 len 12288 data: 8 11 13 24/30 Flush 25/30 Write block 214 len 32768 data: 13 14 15 16 11 11 10 9 26/30 Write block 509 len 4096 data: 12 27/30 Read block 111 len 20480 28/30 Write block 6 len 16384 data: 8 7 6 8 29/30 Write block 337 len 32768 data: 10 11 10 9 10 12 11 10 30/30 Write block 535 len 12288 data: 8 7 7 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 3 Done Done Done false 1005 Acked Write 9 Done Done Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Write 2 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 8 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 24 2 28 Done 24 2 24 50 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1003)) Some(JobId(1023)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:48.649588046Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1733} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 239 239 239 T F T <--- 1 020-039 132 129 132 246 241 246 F F F <--- 2 040-059 121 121 121 222 222 222 F F F 3 060-079 129 129 129 241 241 241 F F F 4 080-099 129 129 129 241 241 241 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 132 128 132 246 237 246 F T F <--- 7 140-159 132 128 132 246 237 246 F T F <--- 8 160-179 126 126 126 233 233 233 F F F 9 180-199 132 132 132 246 245 246 F T F <--- 10 200-219 127 127 127 234 234 234 T F T <--- 11 220-239 131 131 131 244 244 244 T F T <--- 12 240-259 130 130 130 243 243 243 F F F 13 260-279 132 130 132 246 243 246 F F F <--- 14 280-299 132 127 132 246 234 246 F F F <--- 15 300-319 116 116 116 209 209 209 F F F 16 320-339 127 127 127 234 234 234 T F T <--- 17 340-359 131 131 131 244 244 244 T F T <--- 18 360-379 130 130 130 243 243 243 F F F 19 380-399 132 130 132 246 243 246 F F F <--- 20 400-419 124 124 124 227 227 227 F F F 21 420-439 130 130 130 243 243 243 F F F 22 440-459 130 130 130 243 243 243 F F F 23 460-479 130 130 130 243 243 243 F F F 24 480-499 129 129 129 241 241 241 F F F 25 500-519 132 124 132 246 229 246 T F T <--- 26 520-539 128 128 128 240 240 240 T F T <--- 27 540-559 128 128 128 240 240 240 F F F 28 560-579 130 130 130 243 243 243 F F F 29 580-599 132 132 132 245 245 245 F F F Max gen: 132, Max flush: 246 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 133 133 133 247 247 247 F F F 1 020-039 132 132 132 246 246 246 F F F 2 040-059 121 121 121 222 222 222 F F F 3 060-079 129 129 129 241 241 241 F F F 4 080-099 129 129 129 241 241 241 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 132 132 132 246 246 246 F F F 7 140-159 132 132 132 246 246 246 F F F 8 160-179 126 126 126 233 233 233 F F F 9 180-199 132 132 132 246 246 246 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 133 133 133 247 247 247 F F F 12 240-259 130 130 130 243 243 243 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 133 133 133 247 247 247 F F F 17 340-359 133 133 133 247 247 247 F F F 18 360-379 130 130 130 243 243 243 F F F 19 380-399 132 132 132 246 246 246 F F F 20 400-419 124 124 124 227 227 227 F F F 21 420-439 130 130 130 243 243 243 F F F 22 440-459 130 130 130 243 243 243 F F F 23 460-479 130 130 130 243 243 243 F F F 24 480-499 129 129 129 241 241 241 F F F 25 500-519 133 133 133 247 247 247 F F F 26 520-539 133 133 133 247 247 247 F F F 27 540-559 128 128 128 240 240 240 F F F 28 560-579 130 130 130 243 243 243 F F F 29 580-599 132 132 132 245 245 245 F F F Max gen: 133, Max flush: 247 Begin loop 67 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.975074948Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:52.985787396Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.985837515Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.996302918Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.996606178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.996645613Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.996657455Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.996675044Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741} {"msg":"Crucible 2bac5610-b881-4683-adc7-d8972fdfc5a5 has session id: 5fcd0fed-aa4b-4e58-8bed-62708a3b8082","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.99701523Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Upstairs opts: Upstairs UUID: 2bac5610-b881-4683-adc7-d8972fdfc5a5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997050358Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Crucible stats registered with UUID: 2bac5610-b881-4683-adc7-d8972fdfc5a5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997066198Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:52.997082198Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997250243Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"io task","client":"0","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997274318Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"io task","client":"2","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997288889Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"io task","client":"1","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997417648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"task":"crutest"} {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997434237Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741} {"msg":"2bac5610-b881-4683-adc7-d8972fdfc5a5 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997455634Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997468296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"client":"0","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997481907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"client":"1","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997495808Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"client":"2","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.99750935Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"io task","client":"0","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997523451Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"io task","client":"2","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997537632Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"io task","client":"1","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"ds_connection connected from Ok(127.0.0.1:60856)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997552553Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"io task","client":"0","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"ds_connection connected from Ok(127.0.0.1:36948)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997567453Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"io task","client":"2","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"ds_connection connected from Ok(127.0.0.1:53799)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.997582014Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"io task","client":"1","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998252023Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"client":"2","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998286671Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"client":"1","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998418459Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"client":"0","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.99859166Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998630515Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"[0]R flush_numbers[0..12]: [247, 246, 222, 241, 241, 241, 246, 246, 233, 246, 247, 247]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.99876641Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"[0]R generation[0..12]: [133, 132, 121, 129, 129, 129, 132, 132, 126, 132, 133, 133]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998783309Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998799859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"[1]R flush_numbers[0..12]: [247, 246, 222, 241, 241, 241, 246, 246, 233, 246, 247, 247]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998815609Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"[1]R generation[0..12]: [133, 132, 121, 129, 129, 129, 132, 132, 126, 132, 133, 133]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998831279Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998847789Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"[2]R flush_numbers[0..12]: [247, 246, 222, 241, 241, 241, 246, 246, 233, 246, 247, 247]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998863539Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"[2]R generation[0..12]: [133, 132, 121, 129, 129, 129, 132, 132, 126, 132, 133, 133]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998878859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998894909Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998910669Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998931186Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Next flush: 248","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998944308Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.99895737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"":"downstairs","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998971341Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998983853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"client":"0","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.998997565Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"client":"1","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.999011206Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"client":"2","":"downstairs","session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:05:52.999025307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.999039778Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"2bac5610-b881-4683-adc7-d8972fdfc5a5 is now active with session: 5fcd0fed-aa4b-4e58-8bed-62708a3b8082","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.999054769Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.999067441Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741,"session_id":"5fcd0fed-aa4b-4e58-8bed-62708a3b8082"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.999080912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741} {"msg":"Activated sub_volume 2bac5610-b881-4683-adc7-d8972fdfc5a5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:52.999092245Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 76 len 36864 data: 10 7 9 11 9 8 8 9 9 02/30 Write block 2 len 20480 data: 7 10 9 9 9 03/30 Read block 438 len 36864 04/30 Read block 387 len 24576 05/30 Read block 558 len 20480 06/30 Read block 432 len 24576 07/30 Read block 193 len 28672 08/30 Read block 148 len 16384 09/30 Read block 591 len 16384 10/30 Write block 482 len 28672 data: 12 13 10 10 7 11 11 11/30 Write block 568 len 32768 data: 9 10 8 8 9 5 6 10 12/30 Read block 429 len 20480 13/30 Write block 147 len 12288 data: 8 9 8 14/30 Read block 311 len 36864 15/30 Flush 16/30 Write block 232 len 40960 data: 10 12 13 16 17 12 15 15 14 9 17/30 Write block 97 len 4096 data: 7 18/30 Read block 358 len 12288 19/30 Write block 246 len 40960 data: 15 18 23 22 23 19 15 14 13 12 20/30 Write block 395 len 4096 data: 11 21/30 Flush 22/30 Write block 446 len 20480 data: 14 13 10 12 9 23/30 Write block 456 len 16384 data: 12 11 11 14 24/30 Write block 417 len 8192 data: 6 11 25/30 Flush 26/30 Write block 338 len 4096 data: 12 27/30 Write block 238 len 20480 data: 16 16 15 10 8 28/30 Write block 437 len 24576 data: 12 11 11 12 13 16 29/30 Write block 20 len 24576 data: 11 12 12 8 7 6 30/30 Read block 475 len 8192 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 5 Sent Done Done false 1002 Acked Read 9 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Write 7 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 9 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Write 1 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Write 6 Sent Done Done false 1029 Acked Read 2 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:53.011160361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1741} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 133 134 134 247 248 248 F F F <--- 1 020-039 132 132 132 246 246 246 F T T <--- 2 040-059 121 121 121 222 222 222 F F F 3 060-079 129 134 134 241 248 248 T F F <--- 4 080-099 129 134 134 241 249 249 T F F <--- 5 100-119 129 129 129 241 241 241 F F F 6 120-139 132 132 132 246 246 246 F F F 7 140-159 132 134 134 246 248 248 F F F <--- 8 160-179 126 126 126 233 233 233 F F F 9 180-199 132 132 132 246 246 246 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 133 134 134 247 249 249 F T T <--- 12 240-259 130 134 134 243 249 249 F T T <--- 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 133 133 133 247 247 247 F T T <--- 17 340-359 133 133 133 247 247 247 F F F 18 360-379 130 130 130 243 243 243 F F F 19 380-399 132 134 134 246 249 249 F F F <--- 20 400-419 124 134 134 227 250 250 F F F <--- 21 420-439 130 130 130 243 243 243 F T T <--- 22 440-459 130 134 134 243 250 250 F T T <--- 23 460-479 130 130 130 243 243 243 F F F 24 480-499 129 134 134 241 248 248 F F F <--- 25 500-519 133 133 133 247 247 247 F F F 26 520-539 133 133 133 247 247 247 F F F 27 540-559 128 128 128 240 240 240 F F F 28 560-579 130 134 134 243 248 248 F F F <--- 29 580-599 132 132 132 245 245 245 F F F Max gen: 134, Max flush: 250 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 248 248 248 F F F 1 020-039 135 135 135 251 251 251 F F F 2 040-059 121 121 121 222 222 222 F F F 3 060-079 134 134 134 248 248 248 F F F 4 080-099 134 134 134 249 249 249 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 132 132 132 246 246 246 F F F 7 140-159 134 134 134 248 248 248 F F F 8 160-179 126 126 126 233 233 233 F F F 9 180-199 132 132 132 246 246 246 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 135 135 135 251 251 251 F F F 12 240-259 135 135 135 251 251 251 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 135 135 135 251 251 251 F F F 17 340-359 133 133 133 247 247 247 F F F 18 360-379 130 130 130 243 243 243 F F F 19 380-399 134 134 134 249 249 249 F F F 20 400-419 134 134 134 250 250 250 F F F 21 420-439 135 135 135 251 251 251 F F F 22 440-459 135 135 135 251 251 251 F F F 23 460-479 130 130 130 243 243 243 F F F 24 480-499 134 134 134 248 248 248 F F F 25 500-519 133 133 133 247 247 247 F F F 26 520-539 133 133 133 247 247 247 F F F 27 540-559 128 128 128 240 240 240 F F F 28 560-579 134 134 134 248 248 248 F F F 29 580-599 132 132 132 245 245 245 F F F Max gen: 135, Max flush: 251 Begin loop 68 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.345476596Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.357307262Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.357654184Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.357698036Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.357709949Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.357727687Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749} {"msg":"Crucible a591512c-606c-4ca7-a0ff-c93e5bc9c555 has session id: 642296c4-eddd-4c63-92aa-58d2f5529630","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358086652Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Upstairs opts: Upstairs UUID: a591512c-606c-4ca7-a0ff-c93e5bc9c555, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.35812135Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Crucible stats registered with UUID: a591512c-606c-4ca7-a0ff-c93e5bc9c555","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.35813758Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:57.358151031Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358301067Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"io task","client":"0","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358324442Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"io task","client":"2","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358339123Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"io task","client":"1","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358441639Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"task":"crutest"} {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358456619Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749} {"msg":"a591512c-606c-4ca7-a0ff-c93e5bc9c555 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358477576Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358490078Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"client":"0","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358506238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"client":"1","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358520659Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"client":"2","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358534251Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"io task","client":"2","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358548482Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"io task","client":"1","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358562553Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"io task","client":"0","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"ds_connection connected from Ok(127.0.0.1:42784)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358603947Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"io task","client":"1","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"ds_connection connected from Ok(127.0.0.1:60396)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358620147Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"io task","client":"0","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"ds_connection connected from Ok(127.0.0.1:37383)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.358635337Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"io task","client":"2","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359383857Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"client":"0","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.35941018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"client":"1","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.35942693Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"client":"2","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359751206Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359765117Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"[0]R flush_numbers[0..12]: [248, 251, 222, 248, 249, 241, 246, 248, 233, 246, 247, 251]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359801684Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"[0]R generation[0..12]: [134, 135, 121, 134, 134, 129, 132, 134, 126, 132, 133, 135]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359818723Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359837641Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"[1]R flush_numbers[0..12]: [248, 251, 222, 248, 249, 241, 246, 248, 233, 246, 247, 251]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359853591Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"[1]R generation[0..12]: [134, 135, 121, 134, 134, 129, 132, 134, 126, 132, 133, 135]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359869202Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359885431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"[2]R flush_numbers[0..12]: [248, 251, 222, 248, 249, 241, 246, 248, 233, 246, 247, 251]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359901132Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"[2]R generation[0..12]: [134, 135, 121, 134, 134, 129, 132, 134, 126, 132, 133, 135]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359916872Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359932872Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359946333Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359960045Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Next flush: 252","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:05:57.359973156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.359989936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"":"downstairs","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.360003767Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.360016509Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"client":"0","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.3600304Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"client":"1","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.36004623Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"client":"2","":"downstairs","session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.360060531Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.360073713Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"a591512c-606c-4ca7-a0ff-c93e5bc9c555 is now active with session: 642296c4-eddd-4c63-92aa-58d2f5529630","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.360091562Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.360104094Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749,"session_id":"642296c4-eddd-4c63-92aa-58d2f5529630"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.360117466Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749} {"msg":"Activated sub_volume a591512c-606c-4ca7-a0ff-c93e5bc9c555","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:05:57.360128898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 500 len 20480 data: 9 10 9 11 10 02/30 Read block 442 len 12288 03/30 Read block 272 len 32768 04/30 Read block 582 len 28672 05/30 Read block 414 len 32768 06/30 Read block 543 len 32768 07/30 Write block 40 len 16384 data: 8 8 8 7 08/30 Read block 591 len 24576 09/30 Read block 427 len 28672 10/30 Flush 11/30 Flush 12/30 Write block 159 len 20480 data: 14 10 7 10 11 13/30 Read block 8 len 16384 14/30 Read block 134 len 16384 15/30 Write block 375 len 20480 data: 11 11 10 12 13 16/30 Write block 90 len 28672 data: 9 9 8 7 6 8 8 17/30 Read block 134 len 12288 18/30 Flush 19/30 Flush 20/30 Write block 509 len 36864 data: 13 11 10 11 8 8 11 9 9 21/30 Read block 332 len 4096 22/30 Write block 237 len 20480 data: 13 17 17 16 11 23/30 Read block 135 len 4096 24/30 Write block 501 len 32768 data: 11 10 12 11 13 15 14 13 25/30 Write block 135 len 8192 data: 15 14 26/30 Write block 68 len 12288 data: 14 15 16 27/30 Read block 467 len 36864 28/30 Write block 389 len 8192 data: 14 14 29/30 Write block 349 len 16384 data: 14 14 10 11 30/30 Read block 86 len 4096 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Read 7 Done Done Done false 1004 Acked Read 8 Done Done Done false 1005 Acked Read 8 Done Done Done false 1006 Acked Write 4 Done Done Done false 1007 Acked Read 6 Done Done Done false 1008 Acked Read 7 Done Done Done false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Write 5 Done Done Sent false 1015 Acked Write 7 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 9 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 21 21 Done 30 30 9 69 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:05:57.371944503Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1749} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 248 248 248 F F F 1 020-039 135 135 135 251 251 251 F F F 2 040-059 136 136 121 252 252 222 F F T <--- 3 060-079 134 134 134 248 248 248 T T F <--- 4 080-099 136 136 134 254 254 249 F F F <--- 5 100-119 129 129 129 241 241 241 F F F 6 120-139 132 132 132 246 246 246 T T F <--- 7 140-159 136 136 134 254 254 248 F F F <--- 8 160-179 136 136 126 254 254 233 F F F <--- 9 180-199 132 132 132 246 246 246 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 135 135 135 251 251 251 T T F <--- 12 240-259 135 135 135 251 251 251 T T F <--- 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 135 135 135 251 251 251 F F F 17 340-359 133 133 133 247 247 247 T T F <--- 18 360-379 136 136 130 254 254 243 F F F <--- 19 380-399 134 134 134 249 249 249 T T F <--- 20 400-419 134 134 134 250 250 250 F F F 21 420-439 135 135 135 251 251 251 F F F 22 440-459 135 135 135 251 251 251 F F F 23 460-479 130 130 130 243 243 243 F F F 24 480-499 134 134 134 248 248 248 F F F 25 500-519 136 136 133 252 252 247 T T T <--- 26 520-539 133 133 133 247 247 247 F F F 27 540-559 128 128 128 240 240 240 F F F 28 560-579 134 134 134 248 248 248 F F F 29 580-599 132 132 132 245 245 245 F F F Max gen: 136, Max flush: 254 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 134 134 134 248 248 248 F F F 1 020-039 135 135 135 251 251 251 F F F 2 040-059 136 136 136 252 252 252 F F F 3 060-079 137 137 137 255 255 255 F F F 4 080-099 136 136 136 254 254 254 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 137 137 137 255 255 255 F F F 7 140-159 136 136 136 254 254 254 F F F 8 160-179 136 136 136 254 254 254 F F F 9 180-199 132 132 132 246 246 246 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 137 137 137 255 255 255 F F F 12 240-259 137 137 137 255 255 255 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 135 135 135 251 251 251 F F F 17 340-359 137 137 137 255 255 255 F F F 18 360-379 136 136 136 254 254 254 F F F 19 380-399 137 137 137 255 255 255 F F F 20 400-419 134 134 134 250 250 250 F F F 21 420-439 135 135 135 251 251 251 F F F 22 440-459 135 135 135 251 251 251 F F F 23 460-479 130 130 130 243 243 243 F F F 24 480-499 134 134 134 248 248 248 F F F 25 500-519 137 137 137 255 255 255 F F F 26 520-539 133 133 133 247 247 247 F F F 27 540-559 128 128 128 240 240 240 F F F 28 560-579 134 134 134 248 248 248 F F F 29 580-599 132 132 132 245 245 245 F F F Max gen: 137, Max flush: 255 Begin loop 69 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.692820065Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.704452025Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.704774472Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.704813358Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.70482608Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.704844038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757} {"msg":"Crucible 60ff92e9-195f-432d-932c-7f4a7e478b12 has session id: 9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.70520802Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Upstairs opts: Upstairs UUID: 60ff92e9-195f-432d-932c-7f4a7e478b12, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705242418Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Crucible stats registered with UUID: 60ff92e9-195f-432d-932c-7f4a7e478b12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705261116Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:01.705274798Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705415539Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"io task","client":"0","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705439774Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"io task","client":"1","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705454345Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"io task","client":"2","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705563117Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"task":"crutest"} {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705578467Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757} {"msg":"60ff92e9-195f-432d-932c-7f4a7e478b12 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705599374Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705612286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"client":"0","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705626347Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"client":"1","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705640078Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"client":"2","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705654049Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"io task","client":"2","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705668271Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"io task","client":"0","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705682422Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"io task","client":"1","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"ds_connection connected from Ok(127.0.0.1:55618)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.70574826Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"io task","client":"0","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"ds_connection connected from Ok(127.0.0.1:41525)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705776233Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"io task","client":"1","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"ds_connection connected from Ok(127.0.0.1:45805)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.705794741Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"io task","client":"2","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.706476063Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"client":"1","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.706502946Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"client":"0","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.706574581Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"client":"2","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.706834797Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.706872714Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"[0]R flush_numbers[0..12]: [248, 251, 252, 255, 254, 241, 255, 254, 254, 246, 247, 255]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.706944708Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"[0]R generation[0..12]: [134, 135, 136, 137, 136, 129, 137, 136, 136, 132, 133, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.706961748Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.706978047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"[1]R flush_numbers[0..12]: [248, 251, 252, 255, 254, 241, 255, 254, 254, 246, 247, 255]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.706993708Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"[1]R generation[0..12]: [134, 135, 136, 137, 136, 129, 137, 136, 136, 132, 133, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707009118Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707025448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"[2]R flush_numbers[0..12]: [248, 251, 252, 255, 254, 241, 255, 254, 254, 246, 247, 255]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707040968Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"[2]R generation[0..12]: [134, 135, 136, 137, 136, 129, 137, 136, 136, 132, 133, 137]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707056528Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707075706Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707089457Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707103958Z"Wait for a query_work_queue command to finish before sending IO ,"hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Next flush: 256","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707123686Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707137137Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"":"downstairs","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707151089Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707163831Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"client":"0","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707177882Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"client":"1","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707191863Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"client":"2","":"downstairs","session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:06:01.707206254Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707221285Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"60ff92e9-195f-432d-932c-7f4a7e478b12 is now active with session: 9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707236335Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707248907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757,"session_id":"9ca11ad0-11a8-4b1f-b0dc-fa6f66052f1f"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707262309Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757} {"msg":"Activated sub_volume 60ff92e9-195f-432d-932c-7f4a7e478b12","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:01.707273882Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 8 len 32768 data: 7 9 6 6 8 8 11 12 02/30 Flush 03/30 Write block 224 len 4096 data: 13 04/30 Write block 183 len 40960 data: 9 11 10 9 10 10 8 9 10 9 05/30 Read block 456 len 24576 06/30 Read block 55 len 24576 07/30 Read block 198 len 4096 08/30 Read block 515 len 36864 09/30 Write block 428 len 8192 data: 7 9 10/30 Flush 11/30 Write block 341 len 40960 data: 11 13 12 11 11 11 12 13 15 15 12/30 Read block 139 len 20480 13/30 Read block 339 len 16384 14/30 Flush 15/30 Write block 39 len 12288 data: 9 9 9 16/30 Read block 559 len 12288 17/30 Write block 572 len 32768 data: 10 6 7 11 11 13 13 12 18/30 Write block 23 len 24576 data: 9 8 7 4 8 11 19/30 Read block 490 len 12288 20/30 Write block 379 len 24576 data: 14 14 12 9 7 6 21/30 Read block 465 len 16384 22/30 Read block 406 len 12288 23/30 Read block 175 len 4096 24/30 Flush 25/30 Write block 151 len 40960 data: 8 10 8 9 11 12 14 16 15 11 26/30 Write block 405 len 8192 data: 11 10 27/30 Write block 77 len 28672 data: 8 10 12 10 9 9 10 28/30 Read block 7 len 16384 29/30 Read block 25 len 24576 30/30 Read block 274 len 36864 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:20 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1010 Acked Write 10 Done Done Sent false 1011 Acked Read 5 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Read 4 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 20 21 Done 20 19 0 39 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) Some(JobId(1009)) Downstairs last five completed: 1009 1008 1007 1006 1005 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:01.720114202Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1757} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 256 256 256 F F F 1 020-039 138 138 135 259 259 251 F F F <--- 2 040-059 138 138 136 259 259 252 F F F <--- 3 060-079 137 137 137 255 255 255 T T F <--- 4 080-099 136 136 136 254 254 254 T T F <--- 5 100-119 129 129 129 241 241 241 F F F 6 120-139 137 137 137 255 255 255 F F F 7 140-159 136 136 136 254 254 254 T T F <--- 8 160-179 136 136 136 254 254 254 T T F <--- 9 180-199 138 138 138 257 257 257 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 138 138 138 257 257 257 F F F 12 240-259 137 137 137 255 255 255 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 135 135 135 251 251 251 F F F 17 340-359 138 138 137 258 258 255 F F F <--- 18 360-379 138 138 136 259 259 254 F F F <--- 19 380-399 138 138 137 259 259 255 F F F <--- 20 400-419 134 134 134 250 250 250 T T F <--- 21 420-439 138 138 138 257 257 257 F F F 22 440-459 135 135 135 251 251 251 F F F 23 460-479 130 130 130 243 243 243 F F F 24 480-499 134 134 134 248 248 248 F F F 25 500-519 137 137 137 255 255 255 F F F 26 520-539 133 133 133 247 247 247 F F F 27 540-559 128 128 128 240 240 240 F F F 28 560-579 138 138 134 259 259 248 F F F <--- 29 580-599 132 132 132 245 245 245 F F F Max gen: 138, Max flush: 259 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 256 256 256 F F F 1 020-039 138 138 138 259 259 259 F F F 2 040-059 138 138 138 259 259 259 F F F 3 060-079 139 139 139 260 260 260 F F F 4 080-099 139 139 139 260 260 260 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 137 137 137 255 255 255 F F F 7 140-159 139 139 139 260 260 260 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 138 138 138 257 257 257 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 138 138 138 257 257 257 F F F 12 240-259 137 137 137 255 255 255 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 135 135 135 251 251 251 F F F 17 340-359 138 138 138 258 258 258 F F F 18 360-379 138 138 138 259 259 259 F F F 19 380-399 138 138 138 259 259 259 F F F 20 400-419 139 139 139 260 260 260 F F F 21 420-439 138 138 138 257 257 257 F F F 22 440-459 135 135 135 251 251 251 F F F 23 460-479 130 130 130 243 243 243 F F F 24 480-499 134 134 134 248 248 248 F F F 25 500-519 137 137 137 255 255 255 F F F 26 520-539 133 133 133 247 247 247 F F F 27 540-559 128 128 128 240 240 240 F F F 28 560-579 138 138 138 259 259 259 F F F 29 580-599 132 132 132 245 245 245 F F F Max gen: 139, Max flush: 260 Begin loop 70 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.038745523Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.050377504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.050707786Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.05074955Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.05076599Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.050784248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765} {"msg":"Crucible 1cf35a46-0b03-4581-858c-2716105736b5 has session id: 9958fb41-464a-4ff6-935b-315c64e1c29e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051121466Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Upstairs opts: Upstairs UUID: 1cf35a46-0b03-4581-858c-2716105736b5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.05114679Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Crucible stats registered with UUID: 1cf35a46-0b03-4581-858c-2716105736b5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051161811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:06.051175003Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051368141Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"io task","client":"0","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051402939Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"io task","client":"2","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051418589Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"io task","client":"1","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051519566Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"task":"crutest"} {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051537205Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765} {"msg":"1cf35a46-0b03-4581-858c-2716105736b5 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051558821Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051571953Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"client":"0","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051585744Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"client":"1","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051602244Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"client":"2","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051616385Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"io task","client":"2","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051630906Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"io task","client":"0","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051645427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"io task","client":"1","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"ds_connection connected from Ok(127.0.0.1:46388)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051690229Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"io task","client":"0","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44799)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051707678Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"io task","client":"2","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"ds_connection connected from Ok(127.0.0.1:57194)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.051850818Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"io task","client":"1","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.052463673Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"client":"1","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.052513391Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"client":"0","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.05253083Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"client":"2","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.052815422Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.052859524Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"[0]R flush_numbers[0..12]: [256, 259, 259, 260, 260, 241, 255, 260, 260, 257, 247, 257]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.05289752Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"[0]R generation[0..12]: [138, 138, 138, 139, 139, 129, 137, 139, 139, 138, 133, 138]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.052915009Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.052932438Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"[1]R flush_numbers[0..12]: [256, 259, 259, 260, 260, 241, 255, 260, 260, 257, 247, 257]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.052951716Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"[1]R generation[0..12]: [138, 138, 138, 139, 139, 129, 137, 139, 139, 138, 133, 138]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.052967546Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.052984106Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"[2]R flush_numbers[0..12]: [256, 259, 259, 260, 260, 241, 255, 260, 260, 257, 247, 257]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053000475Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"[2]R generation[0..12]: [138, 138, 138, 139, 139, 129, 137, 139, 139, 138, 133, 138]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053016605Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053033425Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Max found gen is 140","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:06:06.053047606Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053066714Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Next flush: 261","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053080505Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053093977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"":"downstairs","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053108168Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.0531208Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"client":"0","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053134861Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"client":"1","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053149792Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"client":"2","":"downstairs","session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"All required reconciliation work is completed",Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.05316888Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053182651Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"1cf35a46-0b03-4581-858c-2716105736b5 is now active with session: 9958fb41-464a-4ff6-935b-315c64e1c29e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053197711Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053210513Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765,"session_id":"9958fb41-464a-4ff6-935b-315c64e1c29e"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053224185Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765} {"msg":"Activated sub_volume 1cf35a46-0b03-4581-858c-2716105736b5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:06.053235308Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 539 len 16384 data: 8 8 7 4 03/30 Read block 229 len 40960 04/30 Read block 60 len 16384 05/30 Write block 3 len 40960 data: 11 10 10 10 8 8 10 7 7 9 06/30 Write block 517 len 12288 data: 10 9 11 07/30 Read block 508 len 4096 08/30 Write block 48 len 32768 data: 6 7 7 7 10 11 11 10 09/30 Write block 33 len 32768 data: 13 11 14 14 13 10 10 10 10/30 Read block 582 len 8192 11/30 Read block 419 len 8192 12/30 Read block 533 len 4096 13/30 Read block 80 len 32768 14/30 Read block 130 len 32768 15/30 Write block 125 len 24576 data: 7 6 5 5 4 7 16/30 Write block 331 len 20480 data: 11 7 7 11 12 17/30 Read block 205 len 8192 18/30 Write block 476 len 8192 data: 9 12 19/30 Read block 385 len 12288 20/30 Write block 231 len 16384 data: 7 11 13 14 21/30 Read block 540 len 4096 22/30 Write block 460 len 8192 data: 11 9 23/30 Write block 125 len 16384 data: 8 7 6 6 24/30 Write block 66 len 20480 data: 13 12 15 16 17 25/30 Write block 154 len 24576 data: 10 12 13 15 17 16 26/30 Read block 518 len 24576 27/30 Write block 40 len 24576 data: 11 10 9 8 7 9 28/30 Read block 146 len 36864 29/30 Write block 227 len 24576 data: 10 10 6 8 8 12 30/30 Write block 532 len 32768 data: 13 12 10 9 8 8 8 9 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Done Done Sent false 1002 Acked Read 10 Done Done Sent false 1003 Acked Read 4 Done Done Sent false 1004 Acked Write 10 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Write 8 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Write 2 Done Done Sent false 1018 Acked Read 3 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Write 4 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Read 6 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 29 33 Done 27 27 0 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1000)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:06.064671401Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1765} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 256 256 256 T T F <--- 1 020-039 138 138 138 259 259 259 T T F <--- 2 040-059 138 138 138 259 259 259 T T F <--- 3 060-079 139 139 139 260 260 260 T T F <--- 4 080-099 139 139 139 260 260 260 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 137 137 137 255 255 255 T T F <--- 7 140-159 139 139 139 260 260 260 T T F <--- 8 160-179 139 139 139 260 260 260 F F F 9 180-199 138 138 138 257 257 257 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 138 138 138 257 257 257 T T F <--- 12 240-259 137 137 137 255 255 255 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 135 135 135 251 251 251 T T F <--- 17 340-359 138 138 138 258 258 258 F F F 18 360-379 138 138 138 259 259 259 F F F 19 380-399 138 138 138 259 259 259 F F F 20 400-419 139 139 139 260 260 260 F F F 21 420-439 138 138 138 257 257 257 F F F 22 440-459 135 135 135 251 251 251 F F F 23 460-479 130 130 130 243 243 243 T T F <--- 24 480-499 134 134 134 248 248 248 F F F 25 500-519 137 137 137 255 255 255 T T F <--- 26 520-539 133 133 133 247 247 247 T T F <--- 27 540-559 128 128 128 240 240 240 T T F <--- 28 560-579 138 138 138 259 259 259 F F F 29 580-599 132 132 132 245 245 245 F F F Max gen: 139, Max flush: 260 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 261 261 261 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 140 140 140 261 261 261 F F F 3 060-079 140 140 140 261 261 261 F F F 4 080-099 139 139 139 260 260 260 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 140 140 140 261 261 261 F F F 7 140-159 140 140 140 261 261 261 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 138 138 138 257 257 257 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 140 140 140 261 261 261 F F F 12 240-259 137 137 137 255 255 255 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 140 140 140 261 261 261 F F F 17 340-359 138 138 138 258 258 258 F F F 18 360-379 138 138 138 259 259 259 F F F 19 380-399 138 138 138 259 259 259 F F F 20 400-419 139 139 139 260 260 260 F F F 21 420-439 138 138 138 257 257 257 F F F 22 440-459 135 135 135 251 251 251 F F F 23 460-479 140 140 140 261 261 261 F F F 24 480-499 134 134 134 248 248 248 F F F 25 500-519 140 140 140 261 261 261 F F F 26 520-539 140 140 140 261 261 261 F F F 27 540-559 140 140 140 261 261 261 F F F 28 560-579 138 138 138 259 259 259 F F F 29 580-599 132 132 132 245 245 245 F F F Max gen: 140, Max flush: 261 Begin loop 71 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.391458448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:10.402081173Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.402139416Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.412373765Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.412671018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.412709764Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.412721656Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.412739415Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773} {"msg":"Crucible 789d17d2-fda7-4d07-9d9f-6b7672d08a5b has session id: d4d783ff-e065-41d7-81e3-b69f18b8c321","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413104076Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Upstairs opts: Upstairs UUID: 789d17d2-fda7-4d07-9d9f-6b7672d08a5b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413143111Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Crucible stats registered with UUID: 789d17d2-fda7-4d07-9d9f-6b7672d08a5b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413158771Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:10.413172253Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413316322Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"io task","client":"2","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413341237Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"io task","client":"1","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413356817Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"io task","client":"0","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413475273Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"task":"crutest"} {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413491023Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773} {"msg":"789d17d2-fda7-4d07-9d9f-6b7672d08a5b active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413512259Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413524751Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"client":"0","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413538753Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"client":"1","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413552754Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"client":"2","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413566255Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"io task","client":"2","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413580676Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"io task","client":"1","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"ds_connection connected from Ok(127.0.0.1:35081)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.41360592Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"io task","client":"2","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.413620721Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"io task","client":"0","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"ds_connection connected from Ok(127.0.0.1:56809)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.41363802Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"io task","client":"1","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"ds_connection connected from Ok(127.0.0.1:45437)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.41378154Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"io task","client":"0","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.414358547Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"client":"1","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.414432301Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"client":"2","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.414555574Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"client":"0","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.414717022Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.414755158Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"[0]R flush_numbers[0..12]: [261, 261, 261, 261, 260, 241, 261, 261, 260, 257, 247, 261]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.414924752Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"[0]R generation[0..12]: [140, 140, 140, 140, 139, 129, 140, 140, 139, 138, 133, 140]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.414941991Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.414958431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"[1]R flush_numbers[0..12]: [261, 261, 261, 261, 260, 241, 261, 261, 260, 257, 247, 261]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.414974221Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"[1]R generation[0..12]: [140, 140, 140, 140, 139, 129, 140, 140, 139, 138, 133, 140]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.414989551Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415005801Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"[2]R flush_numbers[0..12]: [261, 261, 261, 261, 260, 241, 261, 261, 260, 257, 247, 261]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415021551Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"[2]R generation[0..12]: [140, 140, 140, 140, 139, 129, 140, 140, 139, 138, 133, 140]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415039859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415056019Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Max found gen is 141","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415069431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Generation requested: 142 >= found:141","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415083502Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Next flush: 262","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415115062Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415128723Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"":"downstairs","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415143154Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415155956Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"client":"0","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415170018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"client":"1","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415184289Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"client":"2","":"downstairs","session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:06:10.415199239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415214939Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"789d17d2-fda7-4d07-9d9f-6b7672d08a5b is now active with session: d4d783ff-e065-41d7-81e3-b69f18b8c321","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.41522996Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415242742Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773,"session_id":"d4d783ff-e065-41d7-81e3-b69f18b8c321"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415256053Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773} {"msg":"Activated sub_volume 789d17d2-fda7-4d07-9d9f-6b7672d08a5b","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:10.415269385Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 447 len 40960 data: 14 11 13 10 10 9 12 11 13 13 02/30 Flush 03/30 Flush 04/30 Flush 05/30 Write block 542 len 40960 data: 5 3 3 5 6 11 12 11 12 10 06/30 Flush 07/30 Write block 1 len 20480 data: 5 8 12 11 11 08/30 Write block 227 len 20480 data: 11 11 7 9 9 09/30 Write block 545 len 24576 data: 6 7 12 13 12 13 10/30 Flush 11/30 Write block 243 len 32768 data: 9 12 14 16 19 24 23 24 12/30 Flush 13/30 Write block 86 len 32768 data: 12 11 8 7 10 10 9 8 14/30 Write block 77 len 40960 data: 9 11 13 11 10 10 11 10 12 13 15/30 Write block 485 len 4096 data: 11 16/30 Read block 350 len 8192 17/30 Write block 47 len 32768 data: 6 7 8 8 8 11 12 12 18/30 Read block 548 len 4096 19/30 Read block 329 len 24576 20/30 Write block 366 len 28672 data: 12 16 16 13 12 11 9 21/30 Write block 579 len 8192 data: 13 14 22/30 Read block 262 len 36864 23/30 Flush 24/30 Write block 421 len 32768 data: 13 11 12 10 10 8 9 8 25/30 Read block 390 len 16384 26/30 Read block 320 len 24576 27/30 Write block 126 len 32768 data: 8 7 7 5 8 8 9 11 28/30 Write block 186 len 40960 data: 10 11 11 9 10 11 10 8 10 13 29/30 Write block 320 len 20480 data: 9 9 8 8 9 30/30 Write block 420 len 20480 data: 11 14 12 13 11 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 10 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Write 6 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 8 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 3 3 36 Done 0 27 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:10.428561922Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1773} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 142 142 261 266 266 F F F <--- 1 020-039 140 140 140 261 261 261 F F F 2 040-059 140 142 142 261 268 268 F F F <--- 3 060-079 140 142 142 261 268 268 F F F <--- 4 080-099 139 142 142 260 268 268 F F F <--- 5 100-119 129 129 129 241 241 241 F F F 6 120-139 140 140 140 261 261 261 F T T <--- 7 140-159 140 140 140 261 261 261 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 138 138 138 257 257 257 F T T <--- 10 200-219 133 133 133 247 247 247 F F F 11 220-239 140 142 142 261 266 266 F F F <--- 12 240-259 137 142 142 255 267 267 F F F <--- 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 140 140 140 261 261 261 F T T <--- 17 340-359 138 138 138 258 258 258 F F F 18 360-379 138 142 142 259 268 268 F F F <--- 19 380-399 138 138 138 259 259 259 F F F 20 400-419 139 139 139 260 260 260 F F F 21 420-439 138 138 138 257 257 257 F T T <--- 22 440-459 135 142 142 251 262 262 F F F <--- 23 460-479 140 140 140 261 261 261 F F F 24 480-499 134 142 142 248 268 268 F F F <--- 25 500-519 140 140 140 261 261 261 F F F 26 520-539 140 140 140 261 261 261 F F F 27 540-559 140 142 142 261 266 266 F F F <--- 28 560-579 138 142 142 259 268 268 F F F <--- 29 580-599 132 142 142 245 268 268 F F F <--- Max gen: 142, Max flush: 268 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 266 266 266 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 142 142 142 268 268 268 F F F 3 060-079 142 142 142 268 268 268 F F F 4 080-099 142 142 142 268 268 268 F F F 5 100-119 129 129 129 241 241 241 F F F 6 120-139 143 143 143 269 269 269 F F F 7 140-159 140 140 140 261 261 261 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 143 143 143 269 269 269 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 142 142 142 266 266 266 F F F 12 240-259 142 142 142 267 267 267 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 143 143 143 269 269 269 F F F 17 340-359 138 138 138 258 258 258 F F F 18 360-379 142 142 142 268 268 268 F F F 19 380-399 138 138 138 259 259 259 F F F 20 400-419 139 139 139 260 260 260 F F F 21 420-439 143 143 143 269 269 269 F F F 22 440-459 142 142 142 262 262 262 F F F 23 460-479 140 140 140 261 261 261 F F F 24 480-499 142 142 142 268 268 268 F F F 25 500-519 140 140 140 261 261 261 F F F 26 520-539 140 140 140 261 261 261 F F F 27 540-559 142 142 142 266 266 266 F F F 28 560-579 142 142 142 268 268 268 F F F 29 580-599 142 142 142 268 268 268 F F F Max gen: 143, Max flush: 269 Begin loop 72 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.759707378Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.77136896Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.771691358Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.771733611Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.771746513Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.771764811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781} {"msg":"Crucible f2fcaecf-152b-481d-918e-2e77fdc524f6 has session id: 91a2f97e-1b0b-4f02-93d2-981f793ee368","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772141255Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Upstairs opts: Upstairs UUID: f2fcaecf-152b-481d-918e-2e77fdc524f6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772175993Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Crucible stats registered with UUID: f2fcaecf-152b-481d-918e-2e77fdc524f6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772191453Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:14.772204685Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772358458Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"io task","client":"2","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772383652Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"io task","client":"0","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772398253Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"io task","client":"1","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772535677Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"task":"crutest"} {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772551677Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781} {"msg":"f2fcaecf-152b-481d-918e-2e77fdc524f6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772572494Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772585436Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"client":"0","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772599357Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"client":"1","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772613268Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"client":"2","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772627209Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"io task","client":"2","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.77264147Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"io task","client":"0","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.77265736Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"io task","client":"1","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"ds_connection connected from Ok(127.0.0.1:61660)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772704831Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"io task","client":"0","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"ds_connection connected from Ok(127.0.0.1:60545)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.772828763Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"io task","client":"1","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"ds_connection connected from Ok(127.0.0.1:32904)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.77284873Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"io task","client":"2","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.773385573Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"client":"0","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.773444186Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"client":"1","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.773566539Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"client":"2","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.773742868Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.773771021Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"[0]R flush_numbers[0..12]: [266, 261, 268, 268, 268, 241, 269, 261, 260, 269, 247, 266]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.773927882Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"[0]R generation[0..12]: [142, 140, 142, 142, 142, 129, 143, 140, 139, 143, 133, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.773955295Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.773972864Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"[1]R flush_numbers[0..12]: [266, 261, 268, 268, 268, 241, 269, 261, 260, 269, 247, 266]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.773988714Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"[1]R generation[0..12]: [142, 140, 142, 142, 142, 129, 143, 140, 139, 143, 133, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774017646Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774033885Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"[2]R flush_numbers[0..12]: [266, 261, 268, 268, 268, 241, 269, 261, 260, 269, 247, 266]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774049456Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"[2]R generation[0..12]: [142, 140, 142, 142, 142, 129, 143, 140, 139, 143, 133, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774064876Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774086203Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Max found gen is 144","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774099814Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Generation requested: 144 >= found:144","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774113845Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Next flush: 270","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774126907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774139979Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"":"downstairs","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774157048Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.77416988Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"client":"0","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774186619Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"client":"1","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.77420097Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"client":"2","":"downstairs","session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774214912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774228203Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"f2fcaecf-152b-481d-918e-2e77fdc524f6 is now active with session: 91a2f97e-1b0b-4f02-93d2-981f793ee368","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774243074Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774255476Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781,"session_id":"91a2f97e-1b0b-4f02-93d2-981f793ee368"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774269018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781} {"msg":"Activated sub_volume f2fcaecf-152b-481d-918e-2e77fdc524f6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:14.774280261Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 534 len 32768 data: 11 10 9 9 9 10 9 8 02/30 Write block 243 len 40960 data: 10 13 15 17 20 25 24 25 20 16 03/30 Read block 436 len 36864 04/30 Write block 458 len 4096 data: 12 05/30 Read block 524 len 28672 06/30 Read block 545 len 28672 07/30 Read block 117 len 20480 08/30 Read block 237 len 28672 09/30 Read block 275 len 36864 10/30 Write block 510 len 24576 data: 12 11 12 9 9 12 11/30 Read block 394 len 32768 12/30 Flush 13/30 Write block 87 len 28672 data: 12 9 8 11 11 10 9 14/30 Write block 458 len 36864 data: 13 15 12 10 7 6 6 6 6 15/30 Read block 100 len 40960 16/30 Read block 303 len 12288 17/30 Read block 146 len 20480 18/30 Read block 595 len 4096 19/30 Write block 254 len 24576 data: 14 13 12 10 10 9 20/30 Read block 12 len 24576 21/30 Write block 114 len 24576 data: 7 7 7 9 8 8 22/30 Read block 452 len 28672 23/30 Flush 24/30 Write block 418 len 32768 data: 12 12 12 15 13 14 12 11 25/30 Read block 306 len 24576 26/30 Write block 139 len 40960 data: 12 11 11 10 11 10 7 9 9 10 27/30 Write block 412 len 28672 data: 7 9 11 8 9 7 13 28/30 Read block 285 len 20480 29/30 Write block 186 len 20480 data: 11 12 12 10 11 30/30 Read block 233 len 32768 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Sent false 1001 Acked Write 10 Done Done Sent false 1002 Acked Read 9 Done Done Sent false 1003 Acked Write 1 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Write 6 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 7 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Read 5 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Write 5 Done Done Sent false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 30 31 Done 30 29 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:14.787694291Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1781} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 266 266 266 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 142 142 142 268 268 268 F F F 3 060-079 142 142 142 268 268 268 F F F 4 080-099 144 144 142 271 271 268 F F F <--- 5 100-119 144 144 129 271 271 241 F F F <--- 6 120-139 143 143 143 269 269 269 T T F <--- 7 140-159 140 140 140 261 261 261 T T F <--- 8 160-179 139 139 139 260 260 260 F F F 9 180-199 143 143 143 269 269 269 T T F <--- 10 200-219 133 133 133 247 247 247 F F F 11 220-239 142 142 142 266 266 266 F F F 12 240-259 144 144 142 271 271 267 F F F <--- 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 143 143 143 269 269 269 F F F 17 340-359 138 138 138 258 258 258 F F F 18 360-379 142 142 142 268 268 268 F F F 19 380-399 138 138 138 259 259 259 F F F 20 400-419 139 139 139 260 260 260 T T F <--- 21 420-439 143 143 143 269 269 269 T T F <--- 22 440-459 144 144 142 271 271 262 F F F <--- 23 460-479 144 144 140 271 271 261 F F F <--- 24 480-499 142 142 142 268 268 268 F F F 25 500-519 144 144 140 270 270 261 F F F <--- 26 520-539 144 144 140 270 270 261 F F F <--- 27 540-559 144 144 142 270 270 266 F F F <--- 28 560-579 142 142 142 268 268 268 F F F 29 580-599 142 142 142 268 268 268 F F F Max gen: 144, Max flush: 271 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 72 Verifying data now No change after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 266 266 266 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 142 142 142 268 268 268 F F F 3 060-079 142 142 142 268 268 268 F F F 4 080-099 144 144 144 271 271 271 F F F 5 100-119 144 144 144 271 271 271 F F F 6 120-139 145 145 145 272 272 272 F F F 7 140-159 145 145 145 272 272 272 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 145 145 145 272 272 272 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 142 142 142 266 266 266 F F F 12 240-259 144 144 144 271 271 271 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 143 143 143 269 269 269 F F F 17 340-359 138 138 138 258 258 258 F F F 18 360-379 142 142 142 268 268 268 F F F 19 380-399 138 138 138 259 259 259 F F F 20 400-419 145 145 145 272 272 272 F F F 21 420-439 145 145 145 272 272 272 F F F 22 440-459 144 144 144 271 271 271 F F F 23 460-479 144 144 144 271 271 271 F F F 24 480-499 142 142 142 268 268 268 F F F 25 500-519 144 144 144 270 270 270 F F F 26 520-539 144 144 144 270 270 270 F F F 27 540-559 144 144 144 270 270 270 F F F 28 560-579 142 142 142 268 268 268 F F F 29 580-599 142 142 142 268 268 268 F F F Max gen: 145, Max flush: 272 Begin loop 73 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.112998897Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:19.123599845Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.123652692Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.133917481Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.134204181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.134243296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.134259296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.134277475Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789} {"msg":"Crucible 7ca28157-4de7-4cca-ae7c-eade594655f6 has session id: 65ced2b1-ffe1-4105-961d-3b79b20523bd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.134615662Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Upstairs opts: Upstairs UUID: 7ca28157-4de7-4cca-ae7c-eade594655f6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.134641836Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Crucible stats registered with UUID: 7ca28157-4de7-4cca-ae7c-eade594655f6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.134656777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:19.134670038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.134849725Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"io task","client":"1","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.13487483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"io task","client":"0","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.13488992Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"io task","client":"2","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.13501816Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"task":"crutest"} {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135034629Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789} {"msg":"7ca28157-4de7-4cca-ae7c-eade594655f6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135051668Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.13506474Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"client":"0","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135078801Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"client":"1","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135092763Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"client":"2","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135108613Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"io task","client":"2","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135122754Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"io task","client":"0","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135136855Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"io task","client":"1","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:33494)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135172732Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"io task","client":"2","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:52722)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135188782Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"io task","client":"1","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:37111)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135203693Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"io task","client":"0","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.135942938Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"client":"1","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136010486Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"client":"0","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136029744Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"client":"2","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136334852Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136367402Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"[0]R flush_numbers[0..12]: [266, 261, 268, 268, 271, 271, 272, 272, 260, 272, 247, 266]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136421518Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"[0]R generation[0..12]: [142, 140, 142, 142, 144, 144, 145, 145, 139, 145, 133, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136439936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136456745Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"[1]R flush_numbers[0..12]: [266, 261, 268, 268, 271, 271, 272, 272, 260, 272, 247, 266]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136472635Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"[1]R generation[0..12]: [142, 140, 142, 142, 144, 144, 145, 145, 139, 145, 133, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136491064Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136507274Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"[2]R flush_numbers[0..12]: [266, 261, 268, 268, 271, 271, 272, 272, 260, 272, 247, 266]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136522854Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"[2]R generation[0..12]: [142, 140, 142, 142, 144, 144, 145, 145, 139, 145, 133, 142]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136538714Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136555154Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Max found gen is 146","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136568975Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id"Wait for a query_work_queue command to finish before sending IO :"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Generation requested: 146 >= found:146","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136589272Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Next flush: 273","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136602864Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136616215Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"":"downstairs","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136630386Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136642968Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"client":"0","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.13665698Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"client":"1","":"downstairs","session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136671141Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.13668781Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136703231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"7ca28157-4de7-4cca-ae7c-eade594655f6 is now active with session: 65ced2b1-ffe1-4105-961d-3b79b20523bd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136718521Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136731113Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789,"session_id":"65ced2b1-ffe1-4105-961d-3b79b20523bd"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136744715Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789} {"msg":"Activated sub_volume 7ca28157-4de7-4cca-ae7c-eade594655f6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:19.136755937Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 235 len 24576 data: 17 18 14 18 18 17 02/30 Write block 576 len 32768 data: 12 14 14 14 15 14 15 17 03/30 Read block 416 len 28672 04/30 Write block 350 len 28672 data: 16 11 12 9 10 10 11 05/30 Flush 06/30 Write block 338 len 36864 data: 13 11 10 12 14 13 12 12 12 07/30 Read block 286 len 32768 08/30 Read block 119 len 40960 09/30 Read block 449 len 32768 10/30 Read block 10 len 36864 11/30 Read block 389 len 32768 12/30 Write block 395 len 40960 data: 12 12 10 9 10 10 12 12 9 8 13/30 Read block 513 len 32768 14/30 Write block 249 len 4096 data: 25 15/30 Read block 218 len 40960 16/30 Write block 550 len 32768 data: 14 11 9 8 9 9 11 12 17/30 Read block 542 len 20480 18/30 Read block 454 len 28672 19/30 Write block 141 len 4096 data: 12 20/30 Write block 485 len 28672 data: 12 8 12 12 9 9 11 21/30 Write block 188 len 24576 data: 13 11 12 12 11 9 22/30 Read block 255 len 4096 23/30 Flush 24/30 Write block 189 len 24576 data: 12 13 13 12 10 11 25/30 Write block 490 len 24576 data: 10 12 12 14 13 14 26/30 Write block 118 len 36864 data: 9 9 8 6 7 8 10 9 9 27/30 Read block 577 len 4096 28/30 Write block 399 len 32768 data: 11 11 13 13 10 9 12 11 29/30 Write block 513 len 12288 data: 10 10 13 30/30 Write block 234 len 40960 data: 15 18 19 15 19 19 18 12 9 11 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 9 Done Done Done false 1006 Acked Read 8 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Read 8 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Read 1 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 6 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 8 Sent Done Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 2 3 29 Done 1 23 22 46 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:19.149709487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1789} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 266 266 266 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 142 142 142 268 268 268 F F F 3 060-079 142 142 142 268 268 268 F F F 4 080-099 144 144 144 271 271 271 F F F 5 100-119 144 144 144 271 271 271 F T T <--- 6 120-139 145 145 145 272 272 272 F T T <--- 7 140-159 145 146 146 272 274 274 F F F <--- 8 160-179 139 139 139 260 260 260 F F F 9 180-199 145 146 146 272 274 274 F T T <--- 10 200-219 133 133 133 247 247 247 F F F 11 220-239 146 146 146 273 273 273 F T T <--- 12 240-259 146 146 146 273 274 274 F T T <--- 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 143 146 146 269 274 274 T F F <--- 17 340-359 146 146 146 273 274 274 T F F <--- 18 360-379 142 142 142 268 268 268 F F F 19 380-399 138 146 146 259 274 274 F T T <--- 20 400-419 145 146 146 272 274 274 F T T <--- 21 420-439 145 145 145 272 272 272 F F F 22 440-459 144 144 144 271 271 271 F F F 23 460-479 144 144 144 271 271 271 F F F 24 480-499 142 146 146 268 274 274 F T T <--- 25 500-519 144 144 144 270 270 270 F T T <--- 26 520-539 144 144 144 270 270 270 F F F 27 540-559 144 146 146 270 274 274 F F F <--- 28 560-579 146 146 146 273 273 273 F F F 29 580-599 146 146 146 273 273 273 F F F Max gen: 146, Max flush: 274 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 73 Verifying data now No change after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 266 266 266 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 142 142 142 268 268 268 F F F 3 060-079 142 142 142 268 268 268 F F F 4 080-099 144 144 144 271 271 271 F F F 5 100-119 147 147 147 275 275 275 F F F 6 120-139 147 147 147 275 275 275 F F F 7 140-159 146 146 146 274 274 274 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 147 147 147 275 275 275 F F F 10 200-219 133 133 133 247 247 247 F F F 11 220-239 147 147 147 275 275 275 F F F 12 240-259 147 147 147 275 275 275 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 146 146 146 274 274 274 F F F 17 340-359 146 146 146 274 274 274 F F F 18 360-379 142 142 142 268 268 268 F F F 19 380-399 147 147 147 275 275 275 F F F 20 400-419 147 147 147 275 275 275 F F F 21 420-439 145 145 145 272 272 272 F F F 22 440-459 144 144 144 271 271 271 F F F 23 460-479 144 144 144 271 271 271 F F F 24 480-499 147 147 147 275 275 275 F F F 25 500-519 147 147 147 275 275 275 F F F 26 520-539 144 144 144 270 270 270 F F F 27 540-559 146 146 146 274 274 274 F F F 28 560-579 146 146 146 273 273 273 F F F 29 580-599 146 146 146 273 273 273 F F F Max gen: 147, Max flush: 275 Begin loop 74 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.49453199Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.506324919Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.506652543Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.506693967Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.506707089Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.506725147Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797} {"msg":"Crucible db952ff6-6940-4d5f-bb83-680a5aaaddf9 has session id: 2818fad0-141b-4c95-bce0-0274575456a9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507074468Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Upstairs opts: Upstairs UUID: db952ff6-6940-4d5f-bb83-680a5aaaddf9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507107647Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Crucible stats registered with UUID: db952ff6-6940-4d5f-bb83-680a5aaaddf9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507123557Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:23.507136759Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507295779Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"io task","client":"1","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507330177Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"io task","client":"0","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507345737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"io task","client":"2","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507457917Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"task":"crutest"} {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507473337Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797} {"msg":"db952ff6-6940-4d5f-bb83-680a5aaaddf9 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507488388Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.50750105Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"client":"0","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507515281Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"client":"1","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507529092Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"client":"2","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507542884Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"io task","client":"2","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507556995Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"io task","client":"0","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507571036Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"io task","client":"1","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:54346)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507586466Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"io task","client":"2","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:50495)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507601377Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"io task","client":"0","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"ds_connection connected from Ok(127.0.0.1:37958)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.507724559Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"io task","client":"1","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.50837536Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"client":"2","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508415655Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"client":"0","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508432724Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"client":"1","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508745678Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508792009Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"[0]R flush_numbers[0..12]: [266, 261, 268, 268, 271, 275, 275, 274, 260, 275, 247, 275]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508850112Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"[0]R generation[0..12]: [142, 140, 142, 142, 144, 147, 147, 146, 139, 147, 133, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508867191Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508883901Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"[1]R flush_numbers[0..12]: [266, 261, 268, 268, 271, 275, 275, 274, 260, 275, 247, 275]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508899541Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"[1]R generation[0..12]: [142, 140, 142, 142, 144, 147, 147, 146, 139, 147, 133, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508915121Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508931301Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"[2]R flush_numbers[0..12]: [266, 261, 268, 268, 271, 275, 275, 274, 260, 275, 247, 275]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508946951Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"[2]R generation[0..12]: [142, 140, 142, 142, 144, 147, 147, 146, 139, 147, 133, 147]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508962821Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508979181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Max found gen is 148","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.508992882Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"Wait for a query_work_queue command to finish before sending IO msg":"Generation requested: 148 >= found:148","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509015398Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Next flush: 276","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.50902906Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509042671Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"":"downstairs","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509056662Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509069474Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"client":"0","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509083695Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"client":"1","":"downstairs","session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509097537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509114276Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509134333Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"db952ff6-6940-4d5f-bb83-680a5aaaddf9 is now active with session: 2818fad0-141b-4c95-bce0-0274575456a9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509149454Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509162196Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797,"session_id":"2818fad0-141b-4c95-bce0-0274575456a9"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.509175787Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797} {"msg":"Activated sub_volume db952ff6-6940-4d5f-bb83-680a5aaaddf9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:23.50918724Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 586 len 16384 data: 11 10 9 11 02/30 Write block 580 len 32768 data: 16 15 16 18 17 12 12 11 03/30 Read block 82 len 28672 04/30 Read block 63 len 28672 05/30 Flush 06/30 Write block 10 len 28672 data: 8 8 10 9 12 13 8 07/30 Write block 220 len 20480 data: 11 10 9 12 14 08/30 Read block 590 len 8192 09/30 Read block 176 len 24576 10/30 Write block 235 len 4096 data: 19 11/30 Read block 233 len 36864 12/30 Read block 239 len 12288 13/30 Read block 134 len 4096 14/30 Read block 260 len 28672 15/30 Write block 428 len 28672 data: 9 10 6 8 11 11 12 16/30 Read block 493 len 40960 17/30 Write block 492 len 24576 data: 13 15 14 15 10 8 18/30 Write block 370 len 20480 data: 13 12 10 9 9 19/30 Read block 261 len 32768 20/30 Read block 218 len 8192 21/30 Write block 209 len 32768 data: 14 12 12 13 17 14 15 16 22/30 Read block 520 len 4096 23/30 Read block 472 len 24576 24/30 Write block 401 len 28672 data: 14 14 11 10 13 12 12 25/30 Write block 351 len 12288 data: 12 13 10 26/30 Read block 277 len 4096 27/30 Read block 192 len 36864 28/30 Write block 102 len 8192 data: 10 12 29/30 Read block 532 len 16384 30/30 Write block 137 len 16384 data: 13 12 13 12 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 7 Done Done Done false 1006 Acked Write 5 Done Done Done false 1007 Acked Read 2 Done Done Done false 1008 Acked Read 6 Done Done Done false 1009 Acked Write 1 Done Done Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Read 3 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Write 7 Done Sent Done false 1024 Acked Write 3 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 20 1 22 Done 24 5 24 53 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) Some(JobId(1004)) Some(JobId(1004)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:23.520522507Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1797} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 142 142 142 266 266 266 T T T 1 020-039 140 140 140 261 261 261 F F F 2 040-059 142 142 142 268 268 268 F F F 3 060-079 142 142 142 268 268 268 F F F 4 080-099 144 144 144 271 271 271 F F F 5 100-119 147 147 147 275 275 275 T F T <--- 6 120-139 147 147 147 275 275 275 T F T <--- 7 140-159 146 146 146 274 274 274 T F T <--- 8 160-179 139 139 139 260 260 260 F F F 9 180-199 147 147 147 275 275 275 F F F 10 200-219 133 133 133 247 247 247 T F T <--- 11 220-239 147 147 147 275 275 275 T T T 12 240-259 147 147 147 275 275 275 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 146 146 146 274 274 274 F F F 17 340-359 146 146 146 274 274 274 T F T <--- 18 360-379 142 142 142 268 268 268 T F T <--- 19 380-399 147 147 147 275 275 275 F F F 20 400-419 147 147 147 275 275 275 T F T <--- 21 420-439 145 145 145 272 272 272 T F T <--- 22 440-459 144 144 144 271 271 271 F F F 23 460-479 144 144 144 271 271 271 F F F 24 480-499 147 147 147 275 275 275 T F T <--- 25 500-519 147 147 147 275 275 275 F F F 26 520-539 144 144 144 270 270 270 F F F 27 540-559 146 146 146 274 274 274 F F F 28 560-579 146 146 146 273 273 273 F F F 29 580-599 148 148 148 276 276 276 F F F Max gen: 148, Max flush: 276 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 149 149 149 277 277 277 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 142 142 142 268 268 268 F F F 3 060-079 142 142 142 268 268 268 F F F 4 080-099 144 144 144 271 271 271 F F F 5 100-119 149 149 149 277 277 277 F F F 6 120-139 149 149 149 277 277 277 F F F 7 140-159 149 149 149 277 277 277 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 147 147 147 275 275 275 F F F 10 200-219 149 149 149 277 277 277 F F F 11 220-239 149 149 149 277 277 277 F F F 12 240-259 147 147 147 275 275 275 F F F 13 260-279 132 132 132 246 246 246 F F F 14 280-299 132 132 132 246 246 246 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 146 146 146 274 274 274 F F F 17 340-359 149 149 149 277 277 277 F F F 18 360-379 149 149 149 277 277 277 F F F 19 380-399 147 147 147 275 275 275 F F F 20 400-419 149 149 149 277 277 277 F F F 21 420-439 149 149 149 277 277 277 F F F 22 440-459 144 144 144 271 271 271 F F F 23 460-479 144 144 144 271 271 271 F F F 24 480-499 149 149 149 277 277 277 F F F 25 500-519 147 147 147 275 275 275 F F F 26 520-539 144 144 144 270 270 270 F F F 27 540-559 146 146 146 274 274 274 F F F 28 560-579 146 146 146 273 273 273 F F F 29 580-599 148 148 148 276 276 276 F F F Max gen: 149, Max flush: 277 Begin loop 75 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.847243075Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:27.857815301Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.857872315Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.868240399Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.868537922Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.868578237Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.868590669Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.868608428Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805} {"msg":"Crucible 0c781d28-f052-46d7-9cf1-c1c3f36b0f45 has session id: 84157acd-1739-4288-b172-ee50d54a1c06","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.868965034Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Upstairs opts: Upstairs UUID: 0c781d28-f052-46d7-9cf1-c1c3f36b0f45, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.868999752Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Crucible stats registered with UUID: 0c781d28-f052-46d7-9cf1-c1c3f36b0f45","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869015702Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:27.869029054Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869184906Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"io task","client":"1","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869211639Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"io task","client":"0","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.86922647Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"io task","client":"2","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869343236Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"task":"crutest"} {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869358717Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805} {"msg":"0c781d28-f052-46d7-9cf1-c1c3f36b0f45 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869378204Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869391126Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"client":"0","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869407436Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"client":"1","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869421457Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"client":"2","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869435319Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"io task","client":"2","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869449669Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"io task","client":"0","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869471536Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"io task","client":"1","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"ds_connection connected from Ok(127.0.0.1:56802)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869519156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"io task","client":"0","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"ds_connection connected from Ok(127.0.0.1:50370)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869676707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"io task","client":"2","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"ds_connection connected from Ok(127.0.0.1:37551)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.869705039Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"io task","client":"1","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870224533Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"client":"2","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870253155Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"client":"1","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.87030795Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"client":"0","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870622492Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870665226Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"[0]R flush_numbers[0..12]: [277, 261, 268, 268, 271, 277, 277, 277, 260, 275, 277, 277]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870736841Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"[0]R generation[0..12]: [149, 140, 142, 142, 144, 149, 149, 149, 139, 147, 149, 149]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.87075417Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870773737Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"[1]R flush_numbers[0..12]: [277, 261, 268, 268, 271, 277, 277, 277, 260, 275, 277, 277]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870789977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"[1]R generation[0..12]: [149, 140, 142, 142, 144, 149, 149, 149, 139, 147, 149, 149]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870805527Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870821867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"[2]R flush_numbers[0..12]: [277, 261, 268, 268, 271, 277, 277, 277, 260, 275, 277, 277]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870837637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"[2]R generation[0..12]: [149, 140, 142, 142, 144, 149, 149, 149, 139, 147, 149, 149]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870853357Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870869407Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Max found gen is 150","v":0,"name":"crucible","level":30,"timeWait for a query_work_queue command to finish before sending IO ":"2026-03-14T05:06:27.870882549Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Generation requested: 150 >= found:150","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870902676Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Next flush: 278","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870916148Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870929689Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"":"downstairs","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870944Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870957042Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"client":"0","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.870970914Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"client":"1","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:06:27.870986604Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"client":"2","":"downstairs","session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.871002813Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.871016355Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"0c781d28-f052-46d7-9cf1-c1c3f36b0f45 is now active with session: 84157acd-1739-4288-b172-ee50d54a1c06","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.871031555Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.871043898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805,"session_id":"84157acd-1739-4288-b172-ee50d54a1c06"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.871057239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805} {"msg":"Activated sub_volume 0c781d28-f052-46d7-9cf1-c1c3f36b0f45","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:27.871068932Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 523 len 24576 data: 10 10 12 13 14 13 02/30 Read block 105 len 24576 03/30 Read block 344 len 40960 04/30 Write block 429 len 8192 data: 11 7 05/30 Write block 91 len 28672 data: 12 11 10 7 9 9 8 06/30 Write block 451 len 32768 data: 11 10 13 12 14 14 12 14 07/30 Write block 158 len 4096 data: 18 08/30 Flush 09/30 Flush 10/30 Write block 447 len 40960 data: 15 12 14 11 12 11 14 13 15 15 11/30 Read block 29 len 28672 12/30 Read block 178 len 36864 13/30 Read block 422 len 12288 14/30 Write block 55 len 24576 data: 11 7 7 8 10 13 15/30 Write block 486 len 12288 data: 9 13 13 16/30 Read block 325 len 4096 17/30 Read block 264 len 28672 18/30 Write block 382 len 36864 data: 10 8 7 5 9 12 14 15 15 19/30 Write block 228 len 32768 data: 12 8 10 10 13 14 16 20 20/30 Flush 21/30 Flush 22/30 Write block 409 len 40960 data: 5 7 7 8 10 12 9 10 8 14 23/30 Write block 253 len 40960 data: 15 15 14 13 11 11 10 8 8 9 24/30 Read block 120 len 24576 25/30 Write block 522 len 8192 data: 9 11 26/30 Read block 57 len 40960 27/30 Write block 284 len 32768 data: 12 13 9 9 10 9 10 15 28/30 Read block 318 len 32768 29/30 Read block 247 len 28672 30/30 Write block 530 len 20480 data: 13 14 14 13 12 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Write 8 Sent Done Done false 1006 Acked Write 1 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Write 6 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Read 1 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Write 9 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Flush 0 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 1 1 29 Done 3 29 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1020)) Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:27.884158236Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1805} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 149 149 149 277 277 277 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 142 150 150 268 280 280 F F F <--- 3 060-079 142 150 150 268 280 280 F F F <--- 4 080-099 144 150 150 271 278 278 F F F <--- 5 100-119 149 149 149 277 277 277 F F F 6 120-139 149 149 149 277 277 277 F F F 7 140-159 149 150 150 277 278 278 F F F <--- 8 160-179 139 139 139 260 260 260 F F F 9 180-199 147 147 147 275 275 275 F F F 10 200-219 149 149 149 277 277 277 F F F 11 220-239 149 150 150 277 280 280 F F F <--- 12 240-259 147 147 147 275 275 275 F T T <--- 13 260-279 132 132 132 246 246 246 F T T <--- 14 280-299 132 132 132 246 246 246 F T T <--- 15 300-319 116 116 116 209 209 209 F F F 16 320-339 146 146 146 274 274 274 F F F 17 340-359 149 149 149 277 277 277 F F F 18 360-379 149 149 149 277 277 277 F F F 19 380-399 147 150 150 275 280 280 F F F <--- 20 400-419 149 149 149 277 277 277 F T T <--- 21 420-439 149 150 150 277 278 278 F F F <--- 22 440-459 144 150 150 271 280 280 F F F <--- 23 460-479 144 144 144 271 271 271 F F F 24 480-499 149 150 150 277 280 280 F F F <--- 25 500-519 147 147 147 275 275 275 F F F 26 520-539 144 150 150 270 278 278 T T T <--- 27 540-559 146 146 146 274 274 274 F F F 28 560-579 146 146 146 273 273 273 F F F 29 580-599 148 148 148 276 276 276 F F F Max gen: 150, Max flush: 280 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 149 149 149 277 277 277 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 150 150 150 280 280 280 F F F 4 080-099 150 150 150 278 278 278 F F F 5 100-119 149 149 149 277 277 277 F F F 6 120-139 149 149 149 277 277 277 F F F 7 140-159 150 150 150 278 278 278 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 147 147 147 275 275 275 F F F 10 200-219 149 149 149 277 277 277 F F F 11 220-239 150 150 150 280 280 280 F F F 12 240-259 151 151 151 281 281 281 F F F 13 260-279 151 151 151 281 281 281 F F F 14 280-299 151 151 151 281 281 281 F F F 15 300-319 116 116 116 209 209 209 F F F 16 320-339 146 146 146 274 274 274 F F F 17 340-359 149 149 149 277 277 277 F F F 18 360-379 149 149 149 277 277 277 F F F 19 380-399 150 150 150 280 280 280 F F F 20 400-419 151 151 151 281 281 281 F F F 21 420-439 150 150 150 278 278 278 F F F 22 440-459 150 150 150 280 280 280 F F F 23 460-479 144 144 144 271 271 271 F F F 24 480-499 150 150 150 280 280 280 F F F 25 500-519 147 147 147 275 275 275 F F F 26 520-539 151 151 151 281 281 281 F F F 27 540-559 146 146 146 274 274 274 F F F 28 560-579 146 146 146 273 273 273 F F F 29 580-599 148 148 148 276 276 276 F F F Max gen: 151, Max flush: 281 Begin loop 76 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.223276054Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.235158467Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.235473499Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.235511155Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.235523308Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.235540867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813} {"msg":"Crucible 0e5ea939-1eb1-4c4c-857d-c3d9beb8a257 has session id: d7e69e9e-fcd0-4207-9f62-47c0bed57d89","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.23588551Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Upstairs opts: Upstairs UUID: 0e5ea939-1eb1-4c4c-857d-c3d9beb8a257, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.235920208Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Crucible stats registered with UUID: 0e5ea939-1eb1-4c4c-857d-c3d9beb8a257","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.235938837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:32.235952458Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236116375Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"io task","client":"0","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236150334Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"io task","client":"1","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236166294Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"io task","client":"2","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236259865Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"task":"crutest"} {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236275325Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813} {"msg":"0e5ea939-1eb1-4c4c-857d-c3d9beb8a257 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236290296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236303467Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"client":"0","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236317628Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"client":"1","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.2363317Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"client":"2","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236345601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"io task","client":"2","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236359862Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"io task","client":"0","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236374233Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"io task","client":"1","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"ds_connection connected from Ok(127.0.0.1:55391)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236433656Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"io task","client":"2","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"ds_connection connected from Ok(127.0.0.1:49596)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236459479Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"io task","client":"1","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"ds_connection connected from Ok(127.0.0.1:38575)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.236563884Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"io task","client":"0","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237257148Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"client":"2","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237297283Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"client":"1","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237335299Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"client":"0","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237611365Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237650571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"[0]R flush_numbers[0..12]: [277, 261, 280, 280, 278, 277, 277, 278, 260, 275, 277, 280]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237711692Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"[0]R generation[0..12]: [149, 140, 150, 150, 150, 149, 149, 150, 139, 147, 149, 150]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237728562Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237745261Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"[1]R flush_numbers[0..12]: [277, 261, 280, 280, 278, 277, 277, 278, 260, 275, 277, 280]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237761201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"[1]R generation[0..12]: [149, 140, 150, 150, 150, 149, 149, 150, 139, 147, 149, 150]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237776981Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237793431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"[2]R flush_numbers[0..12]: [277, 261, 280, 280, 278, 277, 277, 278, 260, 275, 277, 280]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237809481Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"[2]R generation[0..12]: [149, 140, 150, 150, 150, 149, 149, 150, 139, 147, 149, 150]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237825061Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237844129Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:06:32.237857901Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237876399Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Next flush: 282","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237889741Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237903072Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"":"downstairs","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237917723Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237930535Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"client":"0","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237944906Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"client":"1","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Transition from WaitQuorum to Active"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237961595Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"client":"2","":"downstairs","session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237976076Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.237990227Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"0e5ea939-1eb1-4c4c-857d-c3d9beb8a257 is now active with session: d7e69e9e-fcd0-4207-9f62-47c0bed57d89","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.238005458Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.23801836Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813,"session_id":"d7e69e9e-fcd0-4207-9f62-47c0bed57d89"} {"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.238032031Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813} {"msg":"Activated sub_volume 0e5ea939-1eb1-4c4c-857d-c3d9beb8a257","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:32.238043394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 310 len 20480 data: 15 15 17 15 12 02/30 Write block 466 len 8192 data: 7 6 03/30 Read block 410 len 4096 04/30 Write block 260 len 12288 data: 9 9 10 05/30 Write block 325 len 36864 data: 10 11 11 12 10 10 12 8 8 06/30 Write block 394 len 32768 data: 12 13 13 11 10 12 12 15 07/30 Read block 81 len 36864 08/30 Read block 43 len 24576 09/30 Write block 326 len 12288 data: 12 12 13 10/30 Write block 293 len 16384 data: 12 11 10 10 11/30 Read block 156 len 24576 12/30 Flush 13/30 Write block 402 len 28672 data: 15 12 11 14 13 13 8 14/30 Read block 274 len 36864 15/30 Read block 476 len 20480 16/30 Read block 136 len 16384 17/30 Flush 18/30 Write block 471 len 36864 data: 5 6 5 8 6 10 13 14 13 19/30 Write block 369 len 32768 data: 14 14 13 11 10 10 12 12 20/30 Read block 412 len 32768 21/30 Write block 554 len 40960 data: 10 10 12 13 13 11 12 8 6 5 22/30 Read block 335 len 4096 23/30 Flush 24/30 Write block 239 len 16384 data: 20 19 13 10 25/30 Flush 26/30 Write block 502 len 8192 data: 11 13 27/30 Read block 130 len 36864 28/30 Write block 259 len 8192 data: 11 10 29/30 Read block 353 len 28672 30/30 Read block 54 len 16384 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Write 8 Done Done Done false 1006 Acked Read 9 Done Done Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Read 6 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Read 5 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Write 10 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 23 1 24 Done 30 7 29 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:32.250629824Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1813} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 149 149 149 277 277 277 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 150 150 150 280 280 280 F F F 4 080-099 150 150 150 278 278 278 F F F 5 100-119 149 149 149 277 277 277 F F F 6 120-139 149 149 149 277 277 277 F F F 7 140-159 150 150 150 278 278 278 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 147 147 147 275 275 275 F F F 10 200-219 149 149 149 277 277 277 F F F 11 220-239 152 150 152 285 280 285 F F F <--- 12 240-259 152 151 152 285 281 285 T F T <--- 13 260-279 152 151 152 282 281 282 T T T <--- 14 280-299 152 151 152 282 281 282 F F F <--- 15 300-319 152 116 152 282 209 282 F T F <--- 16 320-339 152 146 152 282 274 282 F T F <--- 17 340-359 149 149 149 277 277 277 F F F 18 360-379 152 149 152 284 277 284 F F F <--- 19 380-399 152 150 152 282 280 282 F T F <--- 20 400-419 152 151 152 283 281 283 F T F <--- 21 420-439 150 150 150 278 278 278 F F F 22 440-459 150 150 150 280 280 280 F F F 23 460-479 152 144 152 284 271 284 F T F <--- 24 480-499 150 150 150 280 280 280 F F F 25 500-519 147 147 147 275 275 275 T F T <--- 26 520-539 151 151 151 281 281 281 F F F 27 540-559 152 146 152 284 274 284 F F F <--- 28 560-579 152 146 152 284 273 284 F F F <--- 29 580-599 148 148 148 276 276 276 F F F Max gen: 152, Max flush: 285 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 149 149 149 277 277 277 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 150 150 150 280 280 280 F F F 4 080-099 150 150 150 278 278 278 F F F 5 100-119 149 149 149 277 277 277 F F F 6 120-139 149 149 149 277 277 277 F F F 7 140-159 150 150 150 278 278 278 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 147 147 147 275 275 275 F F F 10 200-219 149 149 149 277 277 277 F F F 11 220-239 152 152 152 285 285 285 F F F 12 240-259 153 153 153 286 286 286 F F F 13 260-279 153 153 153 286 286 286 F F F 14 280-299 152 152 152 282 282 282 F F F 15 300-319 152 152 152 282 282 282 F F F 16 320-339 152 152 152 282 282 282 F F F 17 340-359 149 149 149 277 277 277 F F F 18 360-379 152 152 152 284 284 284 F F F 19 380-399 152 152 152 282 282 282 F F F 20 400-419 152 152 152 283 283 283 F F F 21 420-439 150 150 150 278 278 278 F F F 22 440-459 150 150 150 280 280 280 F F F 23 460-479 152 152 152 284 284 284 F F F 24 480-499 150 150 150 280 280 280 F F F 25 500-519 153 153 153 286 286 286 F F F 26 520-539 151 151 151 281 281 281 F F F 27 540-559 152 152 152 284 284 284 F F F 28 560-579 152 152 152 284 284 284 F F F 29 580-599 148 148 148 276 276 276 F F F Max gen: 153, Max flush: 286 Begin loop 77 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.604543754Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.616863872Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.617218019Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.617259863Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.617273155Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.617292333Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821} {"msg":"Crucible 3d895d5f-a403-4501-a8ac-ee8cd651ba64 has session id: 71c97714-0619-48c7-9d94-a6600be33cfa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.617701346Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Upstairs opts: Upstairs UUID: 3d895d5f-a403-4501-a8ac-ee8cd651ba64, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.617731607Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Crucible stats registered with UUID: 3d895d5f-a403-4501-a8ac-ee8cd651ba64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.617747597Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:36.617761948Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.61796517Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"io task","client":"1","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.617991293Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"io task","client":"0","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618007663Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"io task","client":"2","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.61815546Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"task":"crutest"} {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618173389Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821} {"msg":"3d895d5f-a403-4501-a8ac-ee8cd651ba64 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618201431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618215493Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"client":"0","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618230933Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"client":"1","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618246593Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"client":"2","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618261823Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"io task","client":"2","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618277294Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"io task","client":"0","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618292924Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"io task","client":"1","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"ds_connection connected from Ok(127.0.0.1:52251)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618318648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"io task","client":"0","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"ds_connection connected from Ok(127.0.0.1:37108)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618335137Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"io task","client":"1","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"ds_connection connected from Ok(127.0.0.1:51757)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.618351897Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"io task","client":"2","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619123222Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"client":"0","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619170213Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"client":"1","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619315012Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"client":"2","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619530086Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619578936Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"[0]R flush_numbers[0..12]: [277, 261, 280, 280, 278, 277, 277, 278, 260, 275, 277, 285]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619704547Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"[0]R generation[0..12]: [149, 140, 150, 150, 150, 149, 149, 150, 139, 147, 149, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619723415Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619741633Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"[1]R flush_numbers[0..12]: [277, 261, 280, 280, 278, 277, 277, 278, 260, 275, 277, 285]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619759292Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"[1]R generation[0..12]: [149, 140, 150, 150, 150, 149, 149, 150, 139, 147, 149, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619776651Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.61979463Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"[2]R flush_numbers[0..12]: [277, 261, 280, 280, 278, 277, 277, 278, 260, 275, 277, 285]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619812339Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"[2]R generation[0..12]: [149, 140, 150, 150, 150, 149, 149, 150, 139, 147, 149, 152]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619829978Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619848256Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Max found gen is 154","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619863307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Generation requested: 154 >= found:154","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619884933Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Next flush: 287","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619899984Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619914785Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"":"downstairs","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619930555Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619946775Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"client":"0","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.619962685Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"client":"1","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:06:36.619977995Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"client":"2","":"downstairs","session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.620005917Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.620022167Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"3d895d5f-a403-4501-a8ac-ee8cd651ba64 is now active with session: 71c97714-0619-48c7-9d94-a6600be33cfa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.620039386Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.620053887Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821,"session_id":"71c97714-0619-48c7-9d94-a6600be33cfa"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.620068688Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821} {"msg":"Activated sub_volume 3d895d5f-a403-4501-a8ac-ee8cd651ba64","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:36.62008114Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 237 len 8192 data: 16 20 02/30 Flush 03/30 Write block 62 len 12288 data: 11 10 9 04/30 Read block 327 len 36864 05/30 Write block 500 len 12288 data: 10 12 12 06/30 Write block 0 len 40960 data: 4 6 9 13 12 12 11 9 9 11 07/30 Read block 60 len 16384 08/30 Write block 133 len 36864 data: 12 13 16 15 14 13 14 13 13 09/30 Read block 275 len 40960 10/30 Write block 94 len 40960 data: 8 10 10 9 5 7 7 9 11 13 11/30 Write block 69 len 16384 data: 17 18 15 12 12/30 Read block 573 len 36864 13/30 Read block 468 len 28672 14/30 Read block 250 len 8192 15/30 Read block 91 len 40960 16/30 Write block 512 len 32768 data: 13 11 11 14 10 11 10 12 17/30 Write block 82 len 28672 data: 11 12 11 13 14 13 10 18/30 Read block 499 len 40960 19/30 Write block 102 len 20480 data: 12 14 10 10 11 20/30 Write block 321 len 12288 data: 10 9 9 21/30 Read block 262 len 28672 22/30 Write block 355 len 8192 data: 11 12 23/30 Read block 564 len 32768 24/30 Write block 130 len 40960 data: 9 9 10 13 14 17 16 15 14 15 25/30 Read block 447 len 28672 26/30 Read block 2 len 24576 27/30 Write block 398 len 24576 data: 11 13 13 16 16 13 28/30 Write block 284 len 32768 data: 13 14 10 10 11 10 11 16 29/30 Read block 245 len 24576 30/30 Write block 100 len 40960 data: 8 10 13 15 11 11 12 8 8 9 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 3 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Write 3 Done Done Done false 1005 Acked Write 10 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Read 2 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Read 7 Done Done Sent false 1025 Acked Read 6 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 25 27 Done 27 27 3 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:36.634582537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1821} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 149 149 149 277 277 277 T T F <--- 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 150 150 150 280 280 280 T T T 4 080-099 150 150 150 278 278 278 T T F <--- 5 100-119 149 149 149 277 277 277 T T F <--- 6 120-139 149 149 149 277 277 277 T T F <--- 7 140-159 150 150 150 278 278 278 T T F <--- 8 160-179 139 139 139 260 260 260 F F F 9 180-199 147 147 147 275 275 275 F F F 10 200-219 149 149 149 277 277 277 F F F 11 220-239 154 154 154 287 287 287 F F F 12 240-259 153 153 153 286 286 286 F F F 13 260-279 153 153 153 286 286 286 F F F 14 280-299 152 152 152 282 282 282 T T F <--- 15 300-319 152 152 152 282 282 282 F F F 16 320-339 152 152 152 282 282 282 T T F <--- 17 340-359 149 149 149 277 277 277 T T F <--- 18 360-379 152 152 152 284 284 284 F F F 19 380-399 152 152 152 282 282 282 T T F <--- 20 400-419 152 152 152 283 283 283 T T F <--- 21 420-439 150 150 150 278 278 278 F F F 22 440-459 150 150 150 280 280 280 F F F 23 460-479 152 152 152 284 284 284 F F F 24 480-499 150 150 150 280 280 280 F F F 25 500-519 153 153 153 286 286 286 T T T 26 520-539 151 151 151 281 281 281 F F F 27 540-559 152 152 152 284 284 284 F F F 28 560-579 152 152 152 284 284 284 F F F 29 580-599 148 148 148 276 276 276 F F F Max gen: 154, Max flush: 287 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 77 Verifying data now No change after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 155 155 155 288 288 288 F F F 4 080-099 155 155 155 288 288 288 F F F 5 100-119 155 155 155 288 288 288 F F F 6 120-139 155 155 155 288 288 288 F F F 7 140-159 155 155 155 288 288 288 F F F 8 160-179 139 139 139 260 260 260 F F F 9 180-199 147 147 147 275 275 275 F F F 10 200-219 149 149 149 277 277 277 F F F 11 220-239 154 154 154 287 287 287 F F F 12 240-259 153 153 153 286 286 286 F F F 13 260-279 153 153 153 286 286 286 F F F 14 280-299 155 155 155 288 288 288 F F F 15 300-319 152 152 152 282 282 282 F F F 16 320-339 155 155 155 288 288 288 F F F 17 340-359 155 155 155 288 288 288 F F F 18 360-379 152 152 152 284 284 284 F F F 19 380-399 155 155 155 288 288 288 F F F 20 400-419 155 155 155 288 288 288 F F F 21 420-439 150 150 150 278 278 278 F F F 22 440-459 150 150 150 280 280 280 F F F 23 460-479 152 152 152 284 284 284 F F F 24 480-499 150 150 150 280 280 280 F F F 25 500-519 155 155 155 288 288 288 F F F 26 520-539 151 151 151 281 281 281 F F F 27 540-559 152 152 152 284 284 284 F F F 28 560-579 152 152 152 284 284 284 F F F 29 580-599 148 148 148 276 276 276 F F F Max gen: 155, Max flush: 288 Begin loop 78 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.977979625Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.989645564Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.989976536Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990015252Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990028034Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990045832Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829} {"msg":"Crucible 09fff0d0-7ddd-4dcf-8608-766c9672ffc3 has session id: ffb808cc-3e75-4b5c-9223-308ec57292cd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.99038367Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Upstairs opts: Upstairs UUID: 09fff0d0-7ddd-4dcf-8608-766c9672ffc3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990424804Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Crucible stats registered with UUID: 09fff0d0-7ddd-4dcf-8608-766c9672ffc3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990440774Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:40.990454216Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990649133Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"io task","client":"2","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990683732Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"io task","client":"0","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990699372Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"io task","client":"1","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990786807Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"task":"crutest"} {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990802167Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829} {"msg":"09fff0d0-7ddd-4dcf-8608-766c9672ffc3 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990821485Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990833947Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"client":"0","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990848248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"client":"1","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990862229Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"client":"2","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990875861Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"io task","client":"2","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990890172Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"io task","client":"0","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990904513Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"io task","client":"1","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"ds_connection connected from Ok(127.0.0.1:40516)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990948495Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"io task","client":"2","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"ds_connection connected from Ok(127.0.0.1:33229)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.990964825Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"io task","client":"1","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"ds_connection connected from Ok(127.0.0.1:49213)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.991066261Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"io task","client":"0","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.991643928Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"client":"1","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.991679836Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"client":"0","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.991772657Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"client":"2","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992028107Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992042718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"[0]R flush_numbers[0..12]: [288, 261, 280, 288, 288, 288, 288, 288, 260, 275, 277, 287]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992185418Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"[0]R generation[0..12]: [155, 140, 150, 155, 155, 155, 155, 155, 139, 147, 149, 154]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992202427Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992219017Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"[1]R flush_numbers[0..12]: [288, 261, 280, 288, 288, 288, 288, 288, 260, 275, 277, 287]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992235007Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"[1]R generation[0..12]: [155, 140, 150, 155, 155, 155, 155, 155, 139, 147, 149, 154]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992250707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992266777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"[2]R flush_numbers[0..12]: [288, 261, 280, 288, 288, 288, 288, 288, 260, 275, 277, 287]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992285035Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"[2]R generation[0..12]: [155, 140, 150, 155, 155, 155, 155, 155, 139, 147, 149, 154]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992301005Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992317105Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992330647Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Generation requested: 156 >= found:156","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-03-14T05:06:40.992350104Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Next flush: 289","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992363376Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992376678Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"":"downstairs","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992390459Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992403221Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"client":"0","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992417162Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"client":"1","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992430884Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"client":"2","":"downstairs","session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992444735Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992461135Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"09fff0d0-7ddd-4dcf-8608-766c9672ffc3 is now active with session: ffb808cc-3e75-4b5c-9223-308ec57292cd","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992476135Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992488787Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829,"session_id":"ffb808cc-3e75-4b5c-9223-308ec57292cd"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992503538Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829} {"msg":"Activated sub_volume 09fff0d0-7ddd-4dcf-8608-766c9672ffc3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:40.992515241Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 208 len 36864 data: 15 15 13 13 14 18 15 16 17 02/30 Write block 531 len 4096 data: 15 03/30 Read block 268 len 12288 04/30 Read block 42 len 4096 05/30 Read block 199 len 36864 06/30 Read block 524 len 36864 07/30 Write block 401 len 24576 data: 17 17 14 12 15 14 08/30 Write block 476 len 36864 data: 11 14 15 14 13 13 13 14 11 09/30 Write block 553 len 8192 data: 9 11 10/30 Read block 295 len 16384 11/30 Write block 146 len 36864 data: 10 10 11 9 9 9 11 9 11 12/30 Write block 505 len 40960 data: 14 16 15 14 14 13 12 14 12 12 13/30 Write block 428 len 32768 data: 10 12 8 9 12 12 13 13 14/30 Flush 15/30 Write block 193 len 32768 data: 11 12 14 9 11 8 9 9 16/30 Read block 40 len 12288 17/30 Read block 45 len 20480 18/30 Write block 96 len 8192 data: 11 10 19/30 Write block 174 len 16384 data: 9 10 12 13 20/30 Write block 525 len 40960 data: 13 14 15 14 13 14 16 15 14 13 21/30 Read block 351 len 8192 22/30 Read block 453 len 4096 23/30 Flush 24/30 Write block 402 len 36864 data: 18 15 13 16 15 14 9 6 8 25/30 Flush 26/30 Write block 479 len 24576 data: 15 14 14 14 15 12 27/30 Read block 243 len 16384 28/30 Write block 490 len 28672 data: 11 13 14 16 15 16 11 29/30 Write block 240 len 28672 data: 20 14 11 12 14 16 18 30/30 Read block 238 len 40960 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Read 3 Done Done Sent false 1003 Acked Read 1 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Read 9 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Write 2 Done Done Sent false 1009 Acked Read 4 Done Done Sent false 1010 Acked Write 9 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Read 5 Done Done Sent false 1017 Acked Write 2 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 9 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Write 7 Done Done Sent false 1028 Acked Write 7 Done Done Sent false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 28 29 Done 30 29 2 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:41.005502319Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1829} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 155 155 155 288 288 288 F F F 4 080-099 156 156 155 290 290 288 F F F <--- 5 100-119 155 155 155 288 288 288 F F F 6 120-139 155 155 155 288 288 288 F F F 7 140-159 156 156 155 289 289 288 F F F <--- 8 160-179 156 156 139 290 290 260 F F F <--- 9 180-199 156 156 147 290 290 275 F F F <--- 10 200-219 156 156 149 290 290 277 F F T <--- 11 220-239 154 154 154 287 287 287 F F F 12 240-259 153 153 153 286 286 286 T T F <--- 13 260-279 153 153 153 286 286 286 F F F 14 280-299 155 155 155 288 288 288 F F F 15 300-319 152 152 152 282 282 282 F F F 16 320-339 155 155 155 288 288 288 F F F 17 340-359 155 155 155 288 288 288 F F F 18 360-379 152 152 152 284 284 284 F F F 19 380-399 155 155 155 288 288 288 F F F 20 400-419 156 156 155 291 291 288 F F F <--- 21 420-439 156 156 150 289 289 278 F F F <--- 22 440-459 150 150 150 280 280 280 F F F 23 460-479 156 156 152 289 289 284 T T F <--- 24 480-499 156 156 150 289 289 280 T T F <--- 25 500-519 156 156 155 289 289 288 F F F <--- 26 520-539 156 156 151 290 290 281 F F T <--- 27 540-559 156 156 152 289 289 284 F F F <--- 28 560-579 152 152 152 284 284 284 F F F 29 580-599 148 148 148 276 276 276 F F F Max gen: 156, Max flush: 291 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 155 155 155 288 288 288 F F F 4 080-099 156 156 156 290 290 290 F F F 5 100-119 155 155 155 288 288 288 F F F 6 120-139 155 155 155 288 288 288 F F F 7 140-159 156 156 156 289 289 289 F F F 8 160-179 156 156 156 290 290 290 F F F 9 180-199 156 156 156 290 290 290 F F F 10 200-219 156 156 156 290 290 290 F F F 11 220-239 154 154 154 287 287 287 F F F 12 240-259 157 157 157 292 292 292 F F F 13 260-279 153 153 153 286 286 286 F F F 14 280-299 155 155 155 288 288 288 F F F 15 300-319 152 152 152 282 282 282 F F F 16 320-339 155 155 155 288 288 288 F F F 17 340-359 155 155 155 288 288 288 F F F 18 360-379 152 152 152 284 284 284 F F F 19 380-399 155 155 155 288 288 288 F F F 20 400-419 156 156 156 291 291 291 F F F 21 420-439 156 156 156 289 289 289 F F F 22 440-459 150 150 150 280 280 280 F F F 23 460-479 157 157 157 292 292 292 F F F 24 480-499 157 157 157 292 292 292 F F F 25 500-519 156 156 156 289 289 289 F F F 26 520-539 156 156 156 290 290 290 F F F 27 540-559 156 156 156 289 289 289 F F F 28 560-579 152 152 152 284 284 284 F F F 29 580-599 148 148 148 276 276 276 F F F Max gen: 157, Max flush: 292 Begin loop 79 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.329801249Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.341386579Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.341701121Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.341738048Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.34175047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.341768589Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837} {"msg":"Crucible a6f42751-a429-4649-a261-e9bfeeab6ed0 has session id: 9ec851f8-c0e1-4a27-959f-3e04ca4406ba","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342115401Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Upstairs opts: Upstairs UUID: a6f42751-a429-4649-a261-e9bfeeab6ed0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342142104Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Crucible stats registered with UUID: a6f42751-a429-4649-a261-e9bfeeab6ed0","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342156895Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:45.342169906Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342325789Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"io task","client":"1","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342349853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"io task","client":"0","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342367043Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"io task","client":"2","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342478582Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"task":"crutest"} {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342493563Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837} {"msg":"a6f42751-a429-4649-a261-e9bfeeab6ed0 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342523604Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342537226Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"client":"0","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342551207Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"client":"1","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342564918Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"client":"2","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342595789Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"io task","client":"2","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342638912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"io task","client":"0","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342661717Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"io task","client":"1","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:57285)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342677777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"io task","client":"1","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:36559)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342692878Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"io task","client":"2","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:33659)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.342794944Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"io task","client":"0","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343428406Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"client":"0","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343456208Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"client":"2","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343473247Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"client":"1","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343775827Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343824896Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"[0]R flush_numbers[0..12]: [288, 261, 280, 288, 290, 288, 288, 289, 290, 290, 290, 287]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343863412Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"[0]R generation[0..12]: [155, 140, 150, 155, 156, 155, 155, 156, 156, 156, 156, 154]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343880581Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343897301Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"[1]R flush_numbers[0..12]: [288, 261, 280, 288, 290, 288, 288, 289, 290, 290, 290, 287]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343913241Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"[1]R generation[0..12]: [155, 140, 150, 155, 156, 155, 155, 156, 156, 156, 156, 154]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343928891Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343945211Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"[2]R flush_numbers[0..12]: [288, 261, 280, 288, 290, 288, 288, 289, 290, 290, 290, 287]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343961031Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"[2]R generation[0..12]: [155, 140, 150, 155, 156, 155, 155, 156, 156, 156, 156, 154]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343976381Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.343992491Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Max found gen is 158","vWait for a query_work_queue command to finish before sending IO ":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344023202Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344037563Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Next flush: 293","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344051194Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344066874Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"":"downstairs","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344080946Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344093458Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"client":"0","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344107349Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"client":"1","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.34412127Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"client":"2","":"downstairs","session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344138889Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344152291Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"a6f42751-a429-4649-a261-e9bfeeab6ed0 is now active with session: 9ec851f8-c0e1-4a27-959f-3e04ca4406ba","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344166922Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344179624Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837,"session_id":"9ec851f8-c0e1-4a27-959f-3e04ca4406ba"} {"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344193065Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837} {"msg":"Activated sub_volume a6f42751-a429-4649-a261-e9bfeeab6ed0","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:45.344204658Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 80 len 12288 data: 12 11 12 02/30 Read block 3 len 4096 03/30 Read block 324 len 12288 04/30 Write block 401 len 12288 data: 18 19 16 05/30 Write block 197 len 32768 data: 12 9 10 10 10 9 13 12 06/30 Flush 07/30 Write block 469 len 4096 data: 5 08/30 Read block 78 len 40960 09/30 Flush 10/30 Write block 581 len 20480 data: 16 17 19 18 13 11/30 Read block 309 len 4096 12/30 Write block 274 len 20480 data: 10 11 11 9 8 13/30 Read block 64 len 8192 14/30 Write block 432 len 16384 data: 13 13 14 14 15/30 Read block 186 len 32768 16/30 Write block 202 len 32768 data: 10 14 13 14 11 11 16 16 17/30 Flush 18/30 Write block 211 len 40960 data: 14 15 19 16 17 18 17 12 12 12 19/30 Read block 150 len 8192 20/30 Read block 318 len 16384 21/30 Read block 374 len 8192 22/30 Flush 23/30 Write block 497 len 28672 data: 9 10 12 11 13 13 14 24/30 Write block 230 len 36864 data: 11 11 14 15 17 21 20 17 21 25/30 Read block 480 len 20480 26/30 Read block 15 len 36864 27/30 Read block 71 len 20480 28/30 Write block 186 len 16384 data: 12 13 14 13 29/30 Write block 336 len 4096 data: 12 30/30 Write block 281 len 4096 data: 7 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Read 1 Done Sent Done false 1002 Acked Read 3 Done Sent Done false 1003 Acked Write 3 Done Sent Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 5 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Write 8 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 10 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 4 Sent Sent Sent false 1028 Acked Write 1 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 30 3 36 Done 27 0 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:45.35592948Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1837} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 155 155 155 288 288 288 F F F 4 080-099 158 156 158 293 290 293 F F F <--- 5 100-119 155 155 155 288 288 288 F F F 6 120-139 155 155 155 288 288 288 F F F 7 140-159 156 156 156 289 289 289 F F F 8 160-179 156 156 156 290 290 290 F F F 9 180-199 158 156 158 293 290 293 T F T <--- 10 200-219 158 156 158 296 290 296 F F F <--- 11 220-239 158 154 158 296 287 296 T F T <--- 12 240-259 157 157 157 292 292 292 F F F 13 260-279 158 153 158 295 286 295 F F F <--- 14 280-299 155 155 155 288 288 288 T F T <--- 15 300-319 152 152 152 282 282 282 F F F 16 320-339 155 155 155 288 288 288 T F T <--- 17 340-359 155 155 155 288 288 288 F F F 18 360-379 152 152 152 284 284 284 F F F 19 380-399 155 155 155 288 288 288 F F F 20 400-419 158 156 158 293 291 293 F F F <--- 21 420-439 158 156 158 295 289 295 F F F <--- 22 440-459 150 150 150 280 280 280 F F F 23 460-479 158 157 158 294 292 294 F F F <--- 24 480-499 157 157 157 292 292 292 T F T <--- 25 500-519 156 156 156 289 289 289 T F T <--- 26 520-539 156 156 156 290 290 290 F F F 27 540-559 156 156 156 289 289 289 F F F 28 560-579 152 152 152 284 284 284 F F F 29 580-599 158 148 158 295 276 295 F F F <--- Max gen: 158, Max flush: 296 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 155 155 155 288 288 288 F F F 4 080-099 158 158 158 293 293 293 F F F 5 100-119 155 155 155 288 288 288 F F F 6 120-139 155 155 155 288 288 288 F F F 7 140-159 156 156 156 289 289 289 F F F 8 160-179 156 156 156 290 290 290 F F F 9 180-199 159 159 159 297 297 297 F F F 10 200-219 158 158 158 296 296 296 F F F 11 220-239 159 159 159 297 297 297 F F F 12 240-259 157 157 157 292 292 292 F F F 13 260-279 158 158 158 295 295 295 F F F 14 280-299 159 159 159 297 297 297 F F F 15 300-319 152 152 152 282 282 282 F F F 16 320-339 159 159 159 297 297 297 F F F 17 340-359 155 155 155 288 288 288 F F F 18 360-379 152 152 152 284 284 284 F F F 19 380-399 155 155 155 288 288 288 F F F 20 400-419 158 158 158 293 293 293 F F F 21 420-439 158 158 158 295 295 295 F F F 22 440-459 150 150 150 280 280 280 F F F 23 460-479 158 158 158 294 294 294 F F F 24 480-499 159 159 159 297 297 297 F F F 25 500-519 159 159 159 297 297 297 F F F 26 520-539 156 156 156 290 290 290 F F F 27 540-559 156 156 156 289 289 289 F F F 28 560-579 152 152 152 284 284 284 F F F 29 580-599 158 158 158 295 295 295 F F F Max gen: 159, Max flush: 297 Begin loop 80 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.679839554Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.691441503Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.691752418Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.691790054Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.691802596Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.691823683Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845} {"msg":"Crucible e5966995-e01a-4e99-9b1b-9f7ebd905af6 has session id: 4209212e-f415-4063-95af-841a13ef3798","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692155345Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Upstairs opts: Upstairs UUID: e5966995-e01a-4e99-9b1b-9f7ebd905af6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692190133Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Crucible stats registered with UUID: e5966995-e01a-4e99-9b1b-9f7ebd905af6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692205893Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:49.692219404Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.69235303Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"io task","client":"1","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692378015Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"io task","client":"0","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692392596Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"io task","client":"2","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692527341Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"task":"crutest"} {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692547668Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845} {"msg":"e5966995-e01a-4e99-9b1b-9f7ebd905af6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692570824Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692583396Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"client":"0","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692597447Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"client":"1","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692611408Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"client":"2","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692659608Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"io task","client":"2","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.69268778Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"io task","client":"0","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.69270339Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"io task","client":"1","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"ds_connection connected from Ok(127.0.0.1:41562)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692718571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"io task","client":"2","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"ds_connection connected from Ok(127.0.0.1:41495)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692733951Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"io task","client":"0","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"ds_connection connected from Ok(127.0.0.1:45308)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.692749771Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"io task","client":"1","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.693456327Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"client":"2","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.693485209Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"client":"0","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.693502558Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"client":"1","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.693835529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.69386614Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"[0]R flush_numbers[0..12]: [288, 261, 280, 288, 293, 288, 288, 289, 290, 297, 296, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.693940773Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"[0]R generation[0..12]: [155, 140, 150, 155, 158, 155, 155, 156, 156, 159, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.693958342Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.693975221Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"[1]R flush_numbers[0..12]: [288, 261, 280, 288, 293, 288, 288, 289, 290, 297, 296, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.693991021Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"[1]R generation[0..12]: [155, 140, 150, 155, 158, 155, 155, 156, 156, 159, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694018794Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694038501Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"[2]R flush_numbers[0..12]: [288, 261, 280, 288, 293, 288, 288, 289, 290, 297, 296, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694054231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"[2]R generation[0..12]: [155, 140, 150, 155, 158, 155, 155, 156, 156, 159, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694069872Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694086301Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694106119Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.69412029Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Next flush: 298","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694133731Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694147133Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"":"downstairs","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694160964Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694173576Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"client":"0","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694187378Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"client":"1","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-03-14T05:06:49.694201069Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"client":"2","":"downstairs","session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694217809Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.69423136Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"e5966995-e01a-4e99-9b1b-9f7ebd905af6 is now active with session: 4209212e-f415-4063-95af-841a13ef3798","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.69424776Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694260282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845,"session_id":"4209212e-f415-4063-95af-841a13ef3798"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694273643Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845} {"msg":"Activated sub_volume e5966995-e01a-4e99-9b1b-9f7ebd905af6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:49.694285196Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 336 len 20480 data: 13 11 14 12 11 02/30 Read block 510 len 12288 03/30 Flush 04/30 Write block 346 len 20480 data: 13 13 14 16 17 05/30 Write block 244 len 36864 data: 15 17 19 21 26 26 26 21 17 06/30 Read block 213 len 8192 07/30 Write block 294 len 8192 data: 12 11 08/30 Read block 409 len 28672 09/30 Read block 22 len 32768 10/30 Read block 370 len 8192 11/30 Read block 464 len 8192 12/30 Flush 13/30 Write block 131 len 12288 data: 10 11 14 14/30 Read block 517 len 20480 15/30 Flush 16/30 Write block 247 len 8192 data: 22 27 17/30 Flush 18/30 Write block 69 len 40960 data: 18 19 16 13 11 15 11 11 10 12 19/30 Write block 545 len 32768 data: 7 8 13 14 13 15 12 10 20/30 Read block 552 len 28672 21/30 Read block 264 len 32768 22/30 Write block 457 len 36864 data: 13 15 16 13 11 8 7 7 7 23/30 Write block 460 len 32768 data: 14 12 9 8 8 8 8 7 24/30 Read block 42 len 8192 25/30 Write block 539 len 36864 data: 11 10 9 6 4 4 8 9 14 26/30 Write block 335 len 12288 data: 13 14 12 27/30 Write block 456 len 24576 data: 16 14 16 17 15 13 28/30 Write block 128 len 16384 data: 8 6 10 11 29/30 Read block 97 len 28672 30/30 Write block 341 len 16384 data: 13 15 14 13 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 5 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Read 2 Done Done Done false 1006 Acked Write 2 Done Done Done false 1007 Acked Read 7 Done Done Done false 1008 Acked Read 8 Done Done Done false 1009 Acked Read 2 Done Done Done false 1010 Acked Read 2 Done Done Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 2 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 10 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 19 1 21 Done 26 8 26 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1002)) Some(JobId(1016)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:49.706068461Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1845} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 155 155 155 288 288 288 T F T <--- 4 080-099 158 158 158 293 293 293 F F F 5 100-119 155 155 155 288 288 288 F F F 6 120-139 160 155 160 300 288 300 T F T <--- 7 140-159 156 156 156 289 289 289 F F F 8 160-179 156 156 156 290 290 290 F F F 9 180-199 159 159 159 297 297 297 F F F 10 200-219 158 158 158 296 296 296 F F F 11 220-239 159 159 159 297 297 297 F F F 12 240-259 160 157 160 301 292 301 F T F <--- 13 260-279 158 158 158 295 295 295 F F F 14 280-299 160 159 160 299 297 299 F T F <--- 15 300-319 152 152 152 282 282 282 F F F 16 320-339 160 160 160 298 298 298 T F T <--- 17 340-359 160 160 160 299 298 299 T T T <--- 18 360-379 152 152 152 284 284 284 F F F 19 380-399 155 155 155 288 288 288 F F F 20 400-419 158 158 158 293 293 293 F F F 21 420-439 158 158 158 295 295 295 F F F 22 440-459 150 150 150 280 280 280 T F T <--- 23 460-479 158 158 158 294 294 294 T F T <--- 24 480-499 159 159 159 297 297 297 F F F 25 500-519 159 159 159 297 297 297 F F F 26 520-539 156 156 156 290 290 290 T F T <--- 27 540-559 156 156 156 289 289 289 T F T <--- 28 560-579 152 152 152 284 284 284 F F F 29 580-599 158 158 158 295 295 295 F F F Max gen: 160, Max flush: 301 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 140 140 140 261 261 261 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 161 161 161 302 302 302 F F F 4 080-099 158 158 158 293 293 293 F F F 5 100-119 155 155 155 288 288 288 F F F 6 120-139 161 161 161 302 302 302 F F F 7 140-159 156 156 156 289 289 289 F F F 8 160-179 156 156 156 290 290 290 F F F 9 180-199 159 159 159 297 297 297 F F F 10 200-219 158 158 158 296 296 296 F F F 11 220-239 159 159 159 297 297 297 F F F 12 240-259 160 160 160 301 301 301 F F F 13 260-279 158 158 158 295 295 295 F F F 14 280-299 160 160 160 299 299 299 F F F 15 300-319 152 152 152 282 282 282 F F F 16 320-339 161 161 161 302 302 302 F F F 17 340-359 161 161 161 302 302 302 F F F 18 360-379 152 152 152 284 284 284 F F F 19 380-399 155 155 155 288 288 288 F F F 20 400-419 158 158 158 293 293 293 F F F 21 420-439 158 158 158 295 295 295 F F F 22 440-459 161 161 161 302 302 302 F F F 23 460-479 161 161 161 302 302 302 F F F 24 480-499 159 159 159 297 297 297 F F F 25 500-519 159 159 159 297 297 297 F F F 26 520-539 161 161 161 302 302 302 F F F 27 540-559 161 161 161 302 302 302 F F F 28 560-579 152 152 152 284 284 284 F F F 29 580-599 158 158 158 295 295 295 F F F Max gen: 161, Max flush: 302 Begin loop 81 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.02499002Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.036695354Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037043095Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037084969Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037097821Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.03711556Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853} {"msg":"Crucible 0c42c9a0-41ca-4526-bb60-8f7ba7f8a36d has session id: de31d226-f323-4284-b540-f026efa87194","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037450639Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Upstairs opts: Upstairs UUID: 0c42c9a0-41ca-4526-bb60-8f7ba7f8a36d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037484857Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Crucible stats registered with UUID: 0c42c9a0-41ca-4526-bb60-8f7ba7f8a36d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037500328Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:54.037513669Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037661656Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"io task","client":"1","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037685911Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"io task","client":"0","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.0377044Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"io task","client":"2","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037824824Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"task":"crutest"} {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037840414Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853} {"msg":"0c42c9a0-41ca-4526-bb60-8f7ba7f8a36d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037861761Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037874872Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"client":"0","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037889213Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"client":"1","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037902955Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"client":"2","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037916746Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"io task","client":"2","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037930927Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"io task","client":"0","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037945338Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"io task","client":"1","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"ds_connection connected from Ok(127.0.0.1:40304)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.037985233Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"io task","client":"1","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"ds_connection connected from Ok(127.0.0.1:44399)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.038062185Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"io task","client":"0","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"ds_connection connected from Ok(127.0.0.1:48066)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.038089218Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"io task","client":"2","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.038622483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"client":"0","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.038672161Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"client":"1","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.038805458Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"client":"2","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.038989542Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039044607Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"[0]R flush_numbers[0..12]: [288, 261, 280, 302, 293, 288, 302, 289, 290, 297, 296, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039166491Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"[0]R generation[0..12]: [155, 140, 150, 161, 158, 155, 161, 156, 156, 159, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.03918361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.03920014Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"[1]R flush_numbers[0..12]: [288, 261, 280, 302, 293, 288, 302, 289, 290, 297, 296, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.0392159Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"[1]R generation[0..12]: [155, 140, 150, 161, 158, 155, 161, 156, 156, 159, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.03923202Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039248439Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"[2]R flush_numbers[0..12]: [288, 261, 280, 302, 293, 288, 302, 289, 290, 297, 296, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039263969Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"[2]R generation[0..12]: [155, 140, 150, 161, 158, 155, 161, 156, 156, 159, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.0392796Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039295969Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Max found gen is 162","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039309581Z",Wait for a query_work_queue command to finish before sending IO "hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039330478Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Next flush: 303","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039346398Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039359999Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"":"downstairs","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.03937386Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039386592Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"client":"0","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039401033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"client":"1","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039418332Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"client":"2","":"downstairs","session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039432344Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039445525Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"0c42c9a0-41ca-4526-bb60-8f7ba7f8a36d is now active with session: de31d226-f323-4284-b540-f026efa87194","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039460446Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039473058Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853,"session_id":"de31d226-f323-4284-b540-f026efa87194"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.03948632Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853} {"msg":"Activated sub_volume 0c42c9a0-41ca-4526-bb60-8f7ba7f8a36d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:54.039497473Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 395 len 20480 data: 14 14 12 12 14 02/30 Write block 383 len 12288 data: 9 8 6 03/30 Write block 515 len 24576 data: 15 11 12 11 13 9 04/30 Write block 245 len 32768 data: 18 20 23 28 27 27 22 18 05/30 Write block 436 len 32768 data: 14 13 12 12 13 14 17 16 06/30 Read block 116 len 28672 07/30 Read block 556 len 28672 08/30 Write block 75 len 28672 data: 12 12 11 13 14 13 12 09/30 Write block 25 len 20480 data: 8 5 9 12 12 10/30 Write block 539 len 4096 data: 12 11/30 Read block 150 len 32768 12/30 Write block 542 len 12288 data: 7 5 5 13/30 Read block 12 len 16384 14/30 Read block 35 len 32768 15/30 Write block 116 len 20480 data: 8 10 10 10 9 16/30 Read block 443 len 24576 17/30 Read block 426 len 8192 18/30 Flush 19/30 Write block 176 len 32768 data: 13 14 14 12 10 8 8 10 20/30 Read block 562 len 12288 21/30 Flush 22/30 Write block 350 len 28672 data: 18 13 14 11 11 12 13 23/30 Write block 317 len 20480 data: 8 6 8 10 11 24/30 Write block 96 len 32768 data: 12 11 6 8 9 11 14 16 25/30 Write block 514 len 40960 data: 13 16 12 13 12 14 10 8 10 12 26/30 Read block 259 len 4096 27/30 Read block 574 len 12288 28/30 Read block 439 len 28672 29/30 Read block 411 len 4096 30/30 Write block 572 len 8192 data: 11 7 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Write 6 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Write 8 Done Done Done false 1005 Acked Read 7 Done Done Done false 1006 Acked Read 7 Done Done Done false 1007 Acked Write 7 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Write 5 Done Done Sent false 1015 Acked Read 6 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 23 25 Done 29 29 7 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:54.050957941Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1853} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 162 162 140 303 303 261 F F F <--- 2 040-059 150 150 150 280 280 280 F F F 3 060-079 162 162 161 303 303 302 F F F <--- 4 080-099 162 162 158 303 303 293 T T F <--- 5 100-119 162 162 155 303 303 288 T T F <--- 6 120-139 162 162 161 303 303 302 F F F <--- 7 140-159 156 156 156 289 289 289 F F F 8 160-179 162 162 156 304 304 290 F F F <--- 9 180-199 162 162 159 304 304 297 F F F <--- 10 200-219 158 158 158 296 296 296 F F F 11 220-239 159 159 159 297 297 297 F F F 12 240-259 162 162 160 303 303 301 F F T <--- 13 260-279 158 158 158 295 295 295 F F F 14 280-299 160 160 160 299 299 299 F F F 15 300-319 152 152 152 282 282 282 T T F <--- 16 320-339 161 161 161 302 302 302 T T F <--- 17 340-359 161 161 161 302 302 302 T T F <--- 18 360-379 152 152 152 284 284 284 F F F 19 380-399 162 162 155 303 303 288 F F T <--- 20 400-419 158 158 158 293 293 293 F F F 21 420-439 162 162 158 303 303 295 F F T <--- 22 440-459 162 162 161 303 303 302 F F T <--- 23 460-479 161 161 161 302 302 302 F F F 24 480-499 159 159 159 297 297 297 F F F 25 500-519 162 162 159 303 303 297 T T T <--- 26 520-539 162 162 161 303 303 302 T T T <--- 27 540-559 162 162 161 303 303 302 F F F <--- 28 560-579 152 152 152 284 284 284 T T F <--- 29 580-599 158 158 158 295 295 295 F F F Max gen: 162, Max flush: 304 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 162 162 162 303 303 303 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 162 162 162 303 303 303 F F F 4 080-099 163 163 163 305 305 305 F F F 5 100-119 163 163 163 305 305 305 F F F 6 120-139 162 162 162 303 303 303 F F F 7 140-159 156 156 156 289 289 289 F F F 8 160-179 162 162 162 304 304 304 F F F 9 180-199 162 162 162 304 304 304 F F F 10 200-219 158 158 158 296 296 296 F F F 11 220-239 159 159 159 297 297 297 F F F 12 240-259 162 162 162 303 303 303 F F F 13 260-279 158 158 158 295 295 295 F F F 14 280-299 160 160 160 299 299 299 F F F 15 300-319 163 163 163 305 305 305 F F F 16 320-339 163 163 163 305 305 305 F F F 17 340-359 163 163 163 305 305 305 F F F 18 360-379 152 152 152 284 284 284 F F F 19 380-399 162 162 162 303 303 303 F F F 20 400-419 158 158 158 293 293 293 F F F 21 420-439 162 162 162 303 303 303 F F F 22 440-459 162 162 162 303 303 303 F F F 23 460-479 161 161 161 302 302 302 F F F 24 480-499 159 159 159 297 297 297 F F F 25 500-519 163 163 163 305 305 305 F F F 26 520-539 163 163 163 305 305 305 F F F 27 540-559 162 162 162 303 303 303 F F F 28 560-579 163 163 163 305 305 305 F F F 29 580-599 158 158 158 295 295 295 F F F Max gen: 163, Max flush: 305 Begin loop 82 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.389350624Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:58.399956579Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.400006328Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.410390372Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.410686646Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.410727201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.410739643Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.410757472Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861} {"msg":"Crucible a3ebe537-e501-48a1-961d-144bde457f59 has session id: 008ecca5-1cd8-47a7-ad5b-0606b1c10657","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.41112549Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Upstairs opts: Upstairs UUID: a3ebe537-e501-48a1-961d-144bde457f59, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411153453Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Crucible stats registered with UUID: a3ebe537-e501-48a1-961d-144bde457f59","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411169163Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:58.411182465Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411325645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"io task","client":"2","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411349799Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"io task","client":"0","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.41136488Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"io task","client":"1","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411504512Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"task":"crutest"} {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411528717Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861} {"msg":"a3ebe537-e501-48a1-961d-144bde457f59 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411541649Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411554411Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"client":"0","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.41157103Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"client":"1","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411585221Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"client":"2","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411614113Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"io task","client":"2","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411659475Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"io task","client":"0","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411674645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"io task","client":"1","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"ds_connection connected from Ok(127.0.0.1:41607)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411689746Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"io task","client":"0","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"ds_connection connected from Ok(127.0.0.1:33220)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411704856Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"io task","client":"2","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"ds_connection connected from Ok(127.0.0.1:61977)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.411719867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"io task","client":"1","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.412486075Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"client":"2","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.412523672Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"client":"0","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.412540671Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"client":"1","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.412856373Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.412870853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"[0]R flush_numbers[0..12]: [288, 303, 280, 303, 305, 305, 303, 289, 304, 304, 296, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.412904682Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"[0]R generation[0..12]: [155, 162, 150, 162, 163, 163, 162, 156, 162, 162, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.412921632Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.41294069Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"[1]R flush_numbers[0..12]: [288, 303, 280, 303, 305, 305, 303, 289, 304, 304, 296, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.412956789Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"[1]R generation[0..12]: [155, 162, 150, 162, 163, 163, 162, 156, 162, 162, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.41297242Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.412988979Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"[2]R flush_numbers[0..12]: [288, 303, 280, 303, 305, 305, 303, 289, 304, 304, 296, 297]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413005009Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"[2]R generation[0..12]: [155, 162, 150, 162, 163, 163, 162, 156, 162, 162, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413020559Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413036859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413050281Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413064352Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Next flush: 306","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413077614Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413091245Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"":"downstairs","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {Wait for a query_work_queue command to finish before sending IO "msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413111382Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413124394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"client":"0","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413138545Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"client":"1","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413154265Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"client":"2","":"downstairs","session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413168626Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413181958Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"a3ebe537-e501-48a1-961d-144bde457f59 is now active with session: 008ecca5-1cd8-47a7-ad5b-0606b1c10657","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413196739Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413212229Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861,"session_id":"008ecca5-1cd8-47a7-ad5b-0606b1c10657"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413225601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861} {"msg":"Activated sub_volume a3ebe537-e501-48a1-961d-144bde457f59","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:06:58.413237143Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 271 len 32768 data: 10 10 8 11 12 12 10 9 02/30 Flush 03/30 Write block 194 len 40960 data: 13 15 10 13 10 11 11 11 11 15 04/30 Read block 206 len 4096 05/30 Write block 365 len 36864 data: 9 13 17 17 15 15 14 12 11 06/30 Write block 190 len 28672 data: 14 14 13 12 14 16 11 07/30 Read block 496 len 24576 08/30 Write block 85 len 8192 data: 14 15 09/30 Read block 508 len 16384 10/30 Read block 531 len 12288 11/30 Write block 428 len 8192 data: 11 13 12/30 Read block 513 len 8192 13/30 Write block 549 len 20480 data: 14 16 13 11 10 14/30 Write block 332 len 8192 data: 9 9 15/30 Read block 44 len 8192 16/30 Write block 231 len 8192 data: 12 15 17/30 Read block 135 len 24576 18/30 Write block 495 len 32768 data: 17 12 10 11 13 12 14 14 19/30 Read block 517 len 20480 20/30 Read block 206 len 36864 21/30 Read block 126 len 20480 22/30 Read block 209 len 40960 23/30 Read block 256 len 28672 24/30 Read block 456 len 16384 25/30 Read block 352 len 32768 26/30 Read block 15 len 8192 27/30 Write block 185 len 4096 data: 11 28/30 Write block 566 len 16384 data: 7 9 10 11 29/30 Write block 262 len 12288 data: 11 11 13 30/30 Read block 7 len 4096 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Write 7 Sent Done Done false 1006 Acked Read 6 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Read 3 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Write 2 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 5 Sent Done Done false 1021 Acked Read 10 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Write 3 Sent Done Sent false 1029 Acked Read 1 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 2 32 Done 0 30 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:06:58.425254222Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1861} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 162 162 162 303 303 303 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 162 162 162 303 303 303 F F F 4 080-099 163 163 163 305 305 305 F T T <--- 5 100-119 163 163 163 305 305 305 F F F 6 120-139 162 162 162 303 303 303 F F F 7 140-159 156 156 156 289 289 289 F F F 8 160-179 162 162 162 304 304 304 F F F 9 180-199 162 162 162 304 304 304 F T T <--- 10 200-219 158 158 158 296 296 296 F T T <--- 11 220-239 159 159 159 297 297 297 F T T <--- 12 240-259 162 162 162 303 303 303 F F F 13 260-279 158 164 164 295 306 306 F T T <--- 14 280-299 160 160 160 299 299 299 F F F 15 300-319 163 163 163 305 305 305 F F F 16 320-339 163 163 163 305 305 305 F T T <--- 17 340-359 163 163 163 305 305 305 F F F 18 360-379 152 152 152 284 284 284 F T T <--- 19 380-399 162 162 162 303 303 303 F F F 20 400-419 158 158 158 293 293 293 F F F 21 420-439 162 162 162 303 303 303 F T T <--- 22 440-459 162 162 162 303 303 303 F F F 23 460-479 161 161 161 302 302 302 F F F 24 480-499 159 159 159 297 297 297 F T T <--- 25 500-519 163 163 163 305 305 305 F T T <--- 26 520-539 163 163 163 305 305 305 F F F 27 540-559 162 162 162 303 303 303 F T T <--- 28 560-579 163 163 163 305 305 305 F T T <--- 29 580-599 158 158 158 295 295 295 F F F Max gen: 164, Max flush: 306 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 162 162 162 303 303 303 F F F 2 040-059 150 150 150 280 280 280 F F F 3 060-079 162 162 162 303 303 303 F F F 4 080-099 165 165 165 307 307 307 F F F 5 100-119 163 163 163 305 305 305 F F F 6 120-139 162 162 162 303 303 303 F F F 7 140-159 156 156 156 289 289 289 F F F 8 160-179 162 162 162 304 304 304 F F F 9 180-199 165 165 165 307 307 307 F F F 10 200-219 165 165 165 307 307 307 F F F 11 220-239 165 165 165 307 307 307 F F F 12 240-259 162 162 162 303 303 303 F F F 13 260-279 165 165 165 307 307 307 F F F 14 280-299 160 160 160 299 299 299 F F F 15 300-319 163 163 163 305 305 305 F F F 16 320-339 165 165 165 307 307 307 F F F 17 340-359 163 163 163 305 305 305 F F F 18 360-379 165 165 165 307 307 307 F F F 19 380-399 162 162 162 303 303 303 F F F 20 400-419 158 158 158 293 293 293 F F F 21 420-439 165 165 165 307 307 307 F F F 22 440-459 162 162 162 303 303 303 F F F 23 460-479 161 161 161 302 302 302 F F F 24 480-499 165 165 165 307 307 307 F F F 25 500-519 165 165 165 307 307 307 F F F 26 520-539 163 163 163 305 305 305 F F F 27 540-559 165 165 165 307 307 307 F F F 28 560-579 165 165 165 307 307 307 F F F 29 580-599 158 158 158 295 295 295 F F F Max gen: 165, Max flush: 307 Begin loop 83 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.759464993Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.771331156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.771644709Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.771681835Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.771693898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.771711956Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869} {"msg":"Crucible 6ee29b2e-c1e7-4058-b681-020847ca05be has session id: c233e185-505b-49e0-bdf4-5876a467c783","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772096695Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Upstairs opts: Upstairs UUID: 6ee29b2e-c1e7-4058-b681-020847ca05be, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772131673Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Crucible stats registered with UUID: 6ee29b2e-c1e7-4058-b681-020847ca05be","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.77215095Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:02.772164642Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772312189Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"io task","client":"2","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772335854Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"io task","client":"0","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772350675Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"io task","client":"1","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772465972Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"task":"crutest"} {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772481223Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869} {"msg":"6ee29b2e-c1e7-4058-b681-020847ca05be active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772511504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772525225Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"client":"0","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772539396Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"client":"1","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772593582Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"client":"2","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772626422Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"io task","client":"2","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772642032Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"io task","client":"0","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772657002Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"io task","client":"1","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"ds_connection connected from Ok(127.0.0.1:43427)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772672263Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"io task","client":"0","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"ds_connection connected from Ok(127.0.0.1:57526)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772686884Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"io task","client":"1","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"ds_connection connected from Ok(127.0.0.1:34553)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.772761487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"io task","client":"2","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773406022Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"client":"0","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773444048Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"client":"1","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773460877Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"client":"2","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773768034Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.77380722Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"[0]R flush_numbers[0..12]: [288, 303, 280, 303, 307, 305, 303, 289, 304, 307, 307, 307]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.77386987Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"[0]R generation[0..12]: [155, 162, 150, 162, 165, 163, 162, 156, 162, 165, 165, 165]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773887099Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773904049Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"[1]R flush_numbers[0..12]: [288, 303, 280, 303, 307, 305, 303, 289, 304, 307, 307, 307]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773919919Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"[1]R generation[0..12]: [155, 162, 150, 162, 165, 163, 162, 156, 162, 165, 165, 165]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773935549Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773951569Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"[2]R flush_numbers[0..12]: [288, 303, 280, 303, 307, 305, 303, 289, 304, 307, 307, 307]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773967239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"[2]R generation[0..12]: [155, 162, 150, 162, 165, 163, 162, 156, 162, 165, 165, 165]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.773982469Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774001707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Max found gen is 166","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774015499Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Generation requested: 166 >= found:166","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.77402973Z",Wait for a query_work_queue command to finish before sending IO "hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Next flush: 308","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774048268Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.77406173Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"":"downstairs","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774076181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774088573Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"client":"0","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774102564Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"client":"1","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774116545Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"client":"2","":"downstairs","session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774130736Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774146656Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"6ee29b2e-c1e7-4058-b681-020847ca05be is now active with session: c233e185-505b-49e0-bdf4-5876a467c783","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774161977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774174799Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869,"session_id":"c233e185-505b-49e0-bdf4-5876a467c783"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.77418814Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869} {"msg":"Activated sub_volume 6ee29b2e-c1e7-4058-b681-020847ca05be","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:02.774199583Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 46 len 36864 data: 4 7 8 9 9 9 12 13 13 02/30 Write block 62 len 12288 data: 12 11 10 03/30 Write block 182 len 36864 data: 9 11 12 12 13 14 15 14 15 04/30 Write block 135 len 36864 data: 18 17 16 15 16 14 14 11 12 05/30 Write block 162 len 28672 data: 11 12 10 11 11 11 10 06/30 Read block 554 len 36864 07/30 Read block 427 len 40960 08/30 Flush 09/30 Write block 391 len 32768 data: 11 11 12 13 15 15 13 13 10/30 Read block 19 len 12288 11/30 Flush 12/30 Write block 219 len 28672 data: 13 13 11 10 13 15 9 13/30 Read block 163 len 20480 14/30 Read block 502 len 32768 15/30 Write block 424 len 24576 data: 13 12 9 10 12 14 16/30 Read block 246 len 12288 17/30 Write block 538 len 20480 data: 10 13 11 10 8 18/30 Write block 100 len 8192 data: 10 12 19/30 Read block 351 len 36864 20/30 Read block 556 len 28672 21/30 Write block 352 len 36864 data: 15 12 12 13 14 11 10 9 12 22/30 Write block 358 len 4096 data: 11 23/30 Read block 415 len 28672 24/30 Read block 529 len 36864 25/30 Read block 280 len 32768 26/30 Write block 566 len 32768 data: 8 10 11 12 9 9 12 8 27/30 Read block 438 len 20480 28/30 Read block 117 len 28672 29/30 Read block 158 len 24576 30/30 Write block 112 len 12288 data: 8 8 8 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Sent false 1001 Acked Write 3 Done Done Sent false 1002 Acked Write 9 Done Done Sent false 1003 Acked Write 9 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Read 9 Done Done Sent false 1006 Acked Read 10 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Write 6 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Write 2 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Read 8 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:02.787211706Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1869} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 162 162 162 303 303 303 F F F 2 040-059 166 166 150 308 308 280 F F F <--- 3 060-079 166 166 162 308 308 303 F F F <--- 4 080-099 165 165 165 307 307 307 F F F 5 100-119 163 163 163 305 305 305 T T F <--- 6 120-139 166 166 162 308 308 303 F F F <--- 7 140-159 166 166 156 308 308 289 F F F <--- 8 160-179 166 166 162 308 308 304 F F F <--- 9 180-199 166 166 165 308 308 307 F F F <--- 10 200-219 165 165 165 307 307 307 T T F <--- 11 220-239 165 165 165 307 307 307 T T F <--- 12 240-259 162 162 162 303 303 303 F F F 13 260-279 165 165 165 307 307 307 F F F 14 280-299 160 160 160 299 299 299 F F F 15 300-319 163 163 163 305 305 305 F F F 16 320-339 165 165 165 307 307 307 F F F 17 340-359 163 163 163 305 305 305 T T F <--- 18 360-379 165 165 165 307 307 307 T T F <--- 19 380-399 166 166 162 309 309 303 F F F <--- 20 400-419 158 158 158 293 293 293 F F F 21 420-439 165 165 165 307 307 307 T T F <--- 22 440-459 162 162 162 303 303 303 F F F 23 460-479 161 161 161 302 302 302 F F F 24 480-499 165 165 165 307 307 307 F F F 25 500-519 165 165 165 307 307 307 F F F 26 520-539 163 163 163 305 305 305 T T F <--- 27 540-559 165 165 165 307 307 307 T T F <--- 28 560-579 165 165 165 307 307 307 T T F <--- 29 580-599 158 158 158 295 295 295 F F F Max gen: 166, Max flush: 309 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 162 162 162 303 303 303 F F F 2 040-059 166 166 166 308 308 308 F F F 3 060-079 166 166 166 308 308 308 F F F 4 080-099 165 165 165 307 307 307 F F F 5 100-119 167 167 167 310 310 310 F F F 6 120-139 166 166 166 308 308 308 F F F 7 140-159 166 166 166 308 308 308 F F F 8 160-179 166 166 166 308 308 308 F F F 9 180-199 166 166 166 308 308 308 F F F 10 200-219 167 167 167 310 310 310 F F F 11 220-239 167 167 167 310 310 310 F F F 12 240-259 162 162 162 303 303 303 F F F 13 260-279 165 165 165 307 307 307 F F F 14 280-299 160 160 160 299 299 299 F F F 15 300-319 163 163 163 305 305 305 F F F 16 320-339 165 165 165 307 307 307 F F F 17 340-359 167 167 167 310 310 310 F F F 18 360-379 167 167 167 310 310 310 F F F 19 380-399 166 166 166 309 309 309 F F F 20 400-419 158 158 158 293 293 293 F F F 21 420-439 167 167 167 310 310 310 F F F 22 440-459 162 162 162 303 303 303 F F F 23 460-479 161 161 161 302 302 302 F F F 24 480-499 165 165 165 307 307 307 F F F 25 500-519 165 165 165 307 307 307 F F F 26 520-539 167 167 167 310 310 310 F F F 27 540-559 167 167 167 310 310 310 F F F 28 560-579 167 167 167 310 310 310 F F F 29 580-599 158 158 158 295 295 295 F F F Max gen: 167, Max flush: 310 Begin loop 84 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.120311215Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.131916022Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.132235181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.132276156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.132289387Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.132307286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877} {"msg":"Crucible dc6a94c7-f0c1-4e1a-89a7-eceeb06f6c08 has session id: 5b552907-45ea-4d4f-b1be-2f747bbeaa54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.132647862Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Upstairs opts: Upstairs UUID: dc6a94c7-f0c1-4e1a-89a7-eceeb06f6c08, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.132674585Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Crucible stats registered with UUID: dc6a94c7-f0c1-4e1a-89a7-eceeb06f6c08","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.132689496Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:07.132702568Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.132866105Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"io task","client":"1","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.132901063Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"io task","client":"0","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.132916603Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"io task","client":"2","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.13303341Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"task":"crutest"} {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.13304845Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877} {"msg":"dc6a94c7-f0c1-4e1a-89a7-eceeb06f6c08 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.133072016Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.133085137Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"client":"0","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.133099198Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"client":"1","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.13311332Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"client":"2","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.133127281Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"io task","client":"2","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.133141572Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"io task","client":"0","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.133155783Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"io task","client":"1","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"ds_connection connected from Ok(127.0.0.1:48447)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.133205342Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"io task","client":"1","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"ds_connection connected from Ok(127.0.0.1:43143)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.133221672Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"io task","client":"2","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"ds_connection connected from Ok(127.0.0.1:54973)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.133236672Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"io task","client":"0","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.13403545Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"client":"0","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.13408344Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"client":"1","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134120587Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"client":"2","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134429782Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134471606Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"[0]R flush_numbers[0..12]: [288, 303, 308, 308, 307, 310, 308, 308, 308, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.13454567Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"[0]R generation[0..12]: [155, 162, 166, 166, 165, 167, 166, 166, 166, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134565447Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134582037Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"[1]R flush_numbers[0..12]: [288, 303, 308, 308, 307, 310, 308, 308, 308, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134597937Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"[1]R generation[0..12]: [155, 162, 166, 166, 165, 167, 166, 166, 166, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134613617Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134630286Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"[2]R flush_numbers[0..12]: [288, 303, 308, 308, 307, 310, 308, 308, 308, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134645867Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"[2]R generation[0..12]: [155, 162, 166, 166, 165, 167, 166, 166, 166, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134661457Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134677747Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Max found gen is 168","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134691408Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","Wait for a query_work_queue command to finish before sending IO pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134711136Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Next flush: 311","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134724757Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134738049Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"":"downstairs","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.13475239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.13476704Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"client":"0","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134781481Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"client":"1","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:07:07.134795253Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"client":"2","":"downstairs","session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134813081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134826573Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"dc6a94c7-f0c1-4e1a-89a7-eceeb06f6c08 is now active with session: 5b552907-45ea-4d4f-b1be-2f747bbeaa54","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134841464Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134854336Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877,"session_id":"5b552907-45ea-4d4f-b1be-2f747bbeaa54"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.134867657Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877} {"msg":"Activated sub_volume dc6a94c7-f0c1-4e1a-89a7-eceeb06f6c08","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:07.13487938Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 401 len 8192 data: 19 20 02/30 Write block 565 len 40960 data: 6 9 11 12 13 10 10 13 9 8 03/30 Write block 412 len 4096 data: 9 04/30 Read block 30 len 8192 05/30 Read block 8 len 28672 06/30 Read block 205 len 16384 07/30 Flush 08/30 Write block 552 len 16384 data: 12 11 12 11 09/30 Write block 520 len 20480 data: 11 9 11 13 11 10/30 Read block 489 len 16384 11/30 Read block 97 len 16384 12/30 Read block 213 len 32768 13/30 Flush 14/30 Write block 432 len 40960 data: 14 14 15 15 15 14 13 13 14 15 15/30 Read block 52 len 8192 16/30 Read block 47 len 36864 17/30 Flush 18/30 Write block 341 len 24576 data: 14 16 15 14 13 14 19/30 Write block 24 len 4096 data: 9 20/30 Read block 237 len 12288 21/30 Read block 515 len 32768 22/30 Read block 30 len 12288 23/30 Read block 211 len 12288 24/30 Read block 446 len 40960 25/30 Flush 26/30 Write block 354 len 24576 data: 13 14 15 12 12 10 27/30 Read block 34 len 4096 28/30 Read block 164 len 36864 29/30 Read block 303 len 32768 30/30 Write block 461 len 32768 data: 14 10 9 9 9 9 8 6 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 10 Done Done Sent false 1002 Acked Write 1 Done Done Sent false 1003 Acked Read 2 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Read 4 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 4 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Read 8 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Write 1 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:07.146651892Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1877} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 168 168 162 314 314 303 F F F <--- 2 040-059 166 166 166 308 308 308 F F F 3 060-079 166 166 166 308 308 308 F F F 4 080-099 165 165 165 307 307 307 F F F 5 100-119 167 167 167 310 310 310 F F F 6 120-139 166 166 166 308 308 308 F F F 7 140-159 166 166 166 308 308 308 F F F 8 160-179 166 166 166 308 308 308 F F F 9 180-199 166 166 166 308 308 308 F F F 10 200-219 167 167 167 310 310 310 F F F 11 220-239 167 167 167 310 310 310 F F F 12 240-259 162 162 162 303 303 303 F F F 13 260-279 165 165 165 307 307 307 F F F 14 280-299 160 160 160 299 299 299 F F F 15 300-319 163 163 163 305 305 305 F F F 16 320-339 165 165 165 307 307 307 F F F 17 340-359 168 168 167 314 314 310 T T F <--- 18 360-379 167 167 167 310 310 310 F F F 19 380-399 166 166 166 309 309 309 F F F 20 400-419 168 168 158 311 311 293 F F T <--- 21 420-439 168 168 167 313 313 310 F F F <--- 22 440-459 168 168 162 313 313 303 F F F <--- 23 460-479 161 161 161 302 302 302 T T F <--- 24 480-499 165 165 165 307 307 307 F F F 25 500-519 165 165 165 307 307 307 F F F 26 520-539 168 168 167 312 312 310 F F F <--- 27 540-559 168 168 167 312 312 310 F F F <--- 28 560-579 168 168 167 311 311 310 F F F <--- 29 580-599 158 158 158 295 295 295 F F F Max gen: 168, Max flush: 314 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 F F F 1 020-039 168 168 168 314 314 314 F F F 2 040-059 166 166 166 308 308 308 F F F 3 060-079 166 166 166 308 308 308 F F F 4 080-099 165 165 165 307 307 307 F F F 5 100-119 167 167 167 310 310 310 F F F 6 120-139 166 166 166 308 308 308 F F F 7 140-159 166 166 166 308 308 308 F F F 8 160-179 166 166 166 308 308 308 F F F 9 180-199 166 166 166 308 308 308 F F F 10 200-219 167 167 167 310 310 310 F F F 11 220-239 167 167 167 310 310 310 F F F 12 240-259 162 162 162 303 303 303 F F F 13 260-279 165 165 165 307 307 307 F F F 14 280-299 160 160 160 299 299 299 F F F 15 300-319 163 163 163 305 305 305 F F F 16 320-339 165 165 165 307 307 307 F F F 17 340-359 169 169 169 315 315 315 F F F 18 360-379 167 167 167 310 310 310 F F F 19 380-399 166 166 166 309 309 309 F F F 20 400-419 168 168 168 311 311 311 F F F 21 420-439 168 168 168 313 313 313 F F F 22 440-459 168 168 168 313 313 313 F F F 23 460-479 169 169 169 315 315 315 F F F 24 480-499 165 165 165 307 307 307 F F F 25 500-519 165 165 165 307 307 307 F F F 26 520-539 168 168 168 312 312 312 F F F 27 540-559 168 168 168 312 312 312 F F F 28 560-579 168 168 168 311 311 311 F F F 29 580-599 158 158 158 295 295 295 F F F Max gen: 169, Max flush: 315 Begin loop 85 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.460289362Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.471897437Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.472212519Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.472249876Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.472261638Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.472279447Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885} {"msg":"Crucible 8959b5fb-28de-47d7-a81d-96c2bb62c655 has session id: 087bfc74-a20e-446d-9640-cee503bd5b65","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.472644538Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Upstairs opts: Upstairs UUID: 8959b5fb-28de-47d7-a81d-96c2bb62c655, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.472682863Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Crucible stats registered with UUID: 8959b5fb-28de-47d7-a81d-96c2bb62c655","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.472698873Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:11.472712315Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.472860282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"io task","client":"0","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.472897409Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"io task","client":"1","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.472912959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"io task","client":"2","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473014075Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"task":"crutest"} {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473029665Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885} {"msg":"8959b5fb-28de-47d7-a81d-96c2bb62c655 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473047174Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473059837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"client":"0","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473073688Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"client":"1","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473087649Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"client":"2","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.47310159Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"io task","client":"2","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473116031Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"io task","client":"0","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473137778Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"io task","client":"1","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"ds_connection connected from Ok(127.0.0.1:64687)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473183199Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"io task","client":"1","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"ds_connection connected from Ok(127.0.0.1:60305)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473201727Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"io task","client":"2","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"ds_connection connected from Ok(127.0.0.1:46785)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473221795Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"io task","client":"0","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473959811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"client":"0","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.473987633Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"client":"1","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474066774Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"client":"2","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474379147Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474393968Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"[0]R flush_numbers[0..12]: [288, 314, 308, 308, 307, 310, 308, 308, 308, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474432464Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"[0]R generation[0..12]: [155, 168, 166, 166, 165, 167, 166, 166, 166, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474449603Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474466453Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"[1]R flush_numbers[0..12]: [288, 314, 308, 308, 307, 310, 308, 308, 308, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474482542Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"[1]R generation[0..12]: [155, 168, 166, 166, 165, 167, 166, 166, 166, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474498093Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474514193Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"[2]R flush_numbers[0..12]: [288, 314, 308, 308, 307, 310, 308, 308, 308, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474529993Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"[2]R generation[0..12]: [155, 168, 166, 166, 165, 167, 166, 166, 166, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474548501Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474564971Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474578542Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"Wait for a query_work_queue command to finish before sending IO ":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474599189Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Next flush: 316","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474612691Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474626142Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"":"downstairs","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474640453Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474652965Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"client":"0","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474667106Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"client":"1","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474681008Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"client":"2","":"downstairs","session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:07:11.474695409Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.47470965Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"8959b5fb-28de-47d7-a81d-96c2bb62c655 is now active with session: 087bfc74-a20e-446d-9640-cee503bd5b65","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.47472486Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474737382Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885,"session_id":"087bfc74-a20e-446d-9640-cee503bd5b65"} {"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474759119Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885} {"msg":"Activated sub_volume 8959b5fb-28de-47d7-a81d-96c2bb62c655","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:11.474772081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 104 len 40960 data: 12 12 13 9 9 10 9 8 9 9 02/30 Read block 402 len 28672 03/30 Write block 333 len 32768 data: 10 12 14 15 13 15 13 12 04/30 Write block 274 len 28672 data: 12 13 13 11 10 9 7 05/30 Read block 443 len 36864 06/30 Read block 556 len 28672 07/30 Read block 572 len 32768 08/30 Flush 09/30 Flush 10/30 Write block 436 len 20480 data: 16 15 14 14 15 11/30 Write block 363 len 36864 data: 10 9 10 14 18 18 16 16 15 12/30 Write block 50 len 40960 data: 10 10 13 14 14 12 8 8 9 11 13/30 Read block 530 len 4096 14/30 Read block 172 len 28672 15/30 Read block 576 len 32768 16/30 Read block 0 len 4096 17/30 Flush 18/30 Write block 75 len 8192 data: 13 13 19/30 Read block 522 len 32768 20/30 Write block 160 len 4096 data: 12 21/30 Read block 466 len 40960 22/30 Read block 79 len 8192 23/30 Flush 24/30 Write block 491 len 28672 data: 14 15 17 16 18 13 11 25/30 Write block 295 len 28672 data: 12 11 9 9 9 8 7 26/30 Read block 216 len 28672 27/30 Read block 118 len 20480 28/30 Write block 6 len 8192 data: 12 10 29/30 Read block 142 len 20480 30/30 Read block 238 len 4096 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Sent false 1001 Acked Read 7 Done Done Sent false 1002 Acked Write 8 Done Done Sent false 1003 Acked Write 7 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 5 Done Done Sent false 1010 Acked Write 9 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 2 Done Done Sent false 1018 Acked Read 8 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Read 10 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 7 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Read 5 Done Done Sent false 1029 Acked Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 30 31 Done 30 29 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:11.487006782Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1885} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 155 155 155 288 288 288 T T F <--- 1 020-039 168 168 168 314 314 314 F F F 2 040-059 170 170 166 318 318 308 F F F <--- 3 060-079 170 170 166 319 319 308 F F F <--- 4 080-099 165 165 165 307 307 307 F F F 5 100-119 170 170 167 316 316 310 F F F <--- 6 120-139 166 166 166 308 308 308 F F F 7 140-159 166 166 166 308 308 308 F F F 8 160-179 170 170 166 319 319 308 F F F <--- 9 180-199 166 166 166 308 308 308 F F F 10 200-219 167 167 167 310 310 310 F F F 11 220-239 167 167 167 310 310 310 F F F 12 240-259 162 162 162 303 303 303 F F F 13 260-279 170 170 165 316 316 307 F F F <--- 14 280-299 170 170 160 316 316 299 T T F <--- 15 300-319 163 163 163 305 305 305 T T F <--- 16 320-339 170 170 165 316 316 307 F F F <--- 17 340-359 170 170 169 316 316 315 F F F <--- 18 360-379 170 170 167 318 318 310 F F F <--- 19 380-399 166 166 166 309 309 309 F F F 20 400-419 168 168 168 311 311 311 F F F 21 420-439 170 170 168 318 318 313 F F F <--- 22 440-459 170 170 168 318 318 313 F F F <--- 23 460-479 169 169 169 315 315 315 F F F 24 480-499 165 165 165 307 307 307 T T F <--- 25 500-519 165 165 165 307 307 307 F F F 26 520-539 168 168 168 312 312 312 F F F 27 540-559 168 168 168 312 312 312 F F F 28 560-579 168 168 168 311 311 311 F F F 29 580-599 158 158 158 295 295 295 F F F Max gen: 170, Max flush: 319 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 171 171 320 320 320 F F F 1 020-039 168 168 168 314 314 314 F F F 2 040-059 170 170 170 318 318 318 F F F 3 060-079 170 170 170 319 319 319 F F F 4 080-099 165 165 165 307 307 307 F F F 5 100-119 170 170 170 316 316 316 F F F 6 120-139 166 166 166 308 308 308 F F F 7 140-159 166 166 166 308 308 308 F F F 8 160-179 170 170 170 319 319 319 F F F 9 180-199 166 166 166 308 308 308 F F F 10 200-219 167 167 167 310 310 310 F F F 11 220-239 167 167 167 310 310 310 F F F 12 240-259 162 162 162 303 303 303 F F F 13 260-279 170 170 170 316 316 316 F F F 14 280-299 171 171 171 320 320 320 F F F 15 300-319 171 171 171 320 320 320 F F F 16 320-339 170 170 170 316 316 316 F F F 17 340-359 170 170 170 316 316 316 F F F 18 360-379 170 170 170 318 318 318 F F F 19 380-399 166 166 166 309 309 309 F F F 20 400-419 168 168 168 311 311 311 F F F 21 420-439 170 170 170 318 318 318 F F F 22 440-459 170 170 170 318 318 318 F F F 23 460-479 169 169 169 315 315 315 F F F 24 480-499 171 171 171 320 320 320 F F F 25 500-519 165 165 165 307 307 307 F F F 26 520-539 168 168 168 312 312 312 F F F 27 540-559 168 168 168 312 312 312 F F F 28 560-579 168 168 168 311 311 311 F F F 29 580-599 158 158 158 295 295 295 F F F Max gen: 171, Max flush: 320 Begin loop 86 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.812197151Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:15.822795751Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.82284369Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.833037455Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.83333185Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.833370455Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.833382528Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.833400127Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893} {"msg":"Crucible c22d119f-485e-4943-aee4-efe7b8ee2aa0 has session id: 0d258709-0325-4961-88b5-9716ced7ef7e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.833754214Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Upstairs opts: Upstairs UUID: c22d119f-485e-4943-aee4-efe7b8ee2aa0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.833788972Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Crucible stats registered with UUID: c22d119f-485e-4943-aee4-efe7b8ee2aa0","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.833804872Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:15.833818004Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.833964772Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"io task","client":"2","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.833989666Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"io task","client":"0","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834008344Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"io task","client":"1","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834144929Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"task":"crutest"} {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834161448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893} {"msg":"c22d119f-485e-4943-aee4-efe7b8ee2aa0 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834191989Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.83420601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"client":"0","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834220621Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"client":"1","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834234632Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"client":"2","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834248743Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"io task","client":"2","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834262904Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"io task","client":"0","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834316611Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"io task","client":"1","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"ds_connection connected from Ok(127.0.0.1:57778)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834374574Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"io task","client":"2","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"ds_connection connected from Ok(127.0.0.1:48325)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834398089Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"io task","client":"0","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"ds_connection connected from Ok(127.0.0.1:61886)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.834456483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"io task","client":"1","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835107144Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"client":"1","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835135296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"client":"2","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835152186Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"client":"0","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835477801Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.83552635Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"[0]R flush_numbers[0..12]: [320, 314, 318, 319, 307, 316, 308, 308, 319, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835570493Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"[0]R generation[0..12]: [171, 168, 170, 170, 165, 170, 166, 166, 170, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835587622Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835604142Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"[1]R flush_numbers[0..12]: [320, 314, 318, 319, 307, 316, 308, 308, 319, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835620012Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"[1]R generation[0..12]: [171, 168, 170, 170, 165, 170, 166, 166, 170, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835635782Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835651981Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"[2]R flush_numbers[0..12]: [320, 314, 318, 319, 307, 316, 308, 308, 319, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835667602Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"[2]R generation[0..12]: [171, 168, 170, 170, 165, 170, 166, 166, 170, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835683282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835699382Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835713253Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Generation requested: 172 >= found:172","v":Wait for a query_work_queue command to finish before sending IO 0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835732401Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Next flush: 321","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835745843Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835761173Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"":"downstairs","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835775444Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835788176Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"client":"0","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835802287Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"client":"1","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835816078Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"client":"2","":"downstairs","session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835830249Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835846369Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"c22d119f-485e-4943-aee4-efe7b8ee2aa0 is now active with session: 0d258709-0325-4961-88b5-9716ced7ef7e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.83586125Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835873812Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893,"session_id":"0d258709-0325-4961-88b5-9716ced7ef7e"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835887254Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893} {"msg":"Activated sub_volume c22d119f-485e-4943-aee4-efe7b8ee2aa0","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:15.835898796Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 169 len 36864 data: 10 9 5 7 7 10 11 14 15 02/30 Flush 03/30 Write block 9 len 28672 data: 12 9 9 11 10 13 14 04/30 Flush 05/30 Write block 147 len 24576 data: 11 12 10 10 10 12 06/30 Write block 300 len 24576 data: 9 8 8 8 8 8 07/30 Flush 08/30 Write block 141 len 24576 data: 15 12 13 11 8 11 09/30 Flush 10/30 Write block 71 len 28672 data: 17 14 12 16 14 14 12 11/30 Read block 422 len 8192 12/30 Read block 215 len 40960 13/30 Read block 272 len 36864 14/30 Write block 482 len 32768 data: 15 16 13 13 10 14 14 10 15/30 Read block 5 len 40960 16/30 Read block 154 len 32768 17/30 Write block 25 len 28672 data: 9 6 10 13 13 13 12 18/30 Write block 33 len 32768 data: 14 12 15 15 14 11 11 12 19/30 Flush 20/30 Write block 327 len 40960 data: 13 14 11 11 13 10 11 13 15 16 21/30 Read block 160 len 8192 22/30 Write block 173 len 12288 data: 8 11 12 23/30 Write block 255 len 24576 data: 15 14 12 12 12 11 24/30 Read block 320 len 32768 25/30 Write block 563 len 8192 data: 6 6 26/30 Write block 483 len 8192 data: 17 14 27/30 Read block 316 len 36864 28/30 Write block 116 len 40960 data: 9 11 11 11 10 7 8 9 11 10 29/30 Read block 141 len 40960 30/30 Write block 428 len 40960 data: 13 15 9 10 15 15 16 16 17 16 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 7 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 6 Sent Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 7 Sent Done Done false 1010 Acked Read 2 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Read 9 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Write 10 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 1 30 Done 0 27 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:15.850355831Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1893} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 172 172 320 322 322 F F F <--- 1 020-039 168 172 172 314 325 325 F F F <--- 2 040-059 170 172 172 318 325 325 F F F <--- 3 060-079 170 172 172 319 325 325 F F F <--- 4 080-099 165 165 165 307 307 307 F F F 5 100-119 170 170 170 316 316 316 F T T <--- 6 120-139 166 166 166 308 308 308 F T T <--- 7 140-159 166 172 172 308 324 324 F F F <--- 8 160-179 172 172 172 321 321 321 F T T <--- 9 180-199 166 166 166 308 308 308 F F F 10 200-219 167 167 167 310 310 310 F F F 11 220-239 167 167 167 310 310 310 F F F 12 240-259 162 162 162 303 303 303 F T T <--- 13 260-279 170 170 170 316 316 316 F T T <--- 14 280-299 171 171 171 320 320 320 F F F 15 300-319 171 172 172 320 323 323 F F F <--- 16 320-339 170 170 170 316 316 316 F T T <--- 17 340-359 170 170 170 316 316 316 F F F 18 360-379 170 170 170 318 318 318 F F F 19 380-399 166 166 166 309 309 309 F F F 20 400-419 168 168 168 311 311 311 F F F 21 420-439 170 170 170 318 318 318 F T T <--- 22 440-459 170 170 170 318 318 318 F F F 23 460-479 169 169 169 315 315 315 F F F 24 480-499 171 172 172 320 325 325 F T T <--- 25 500-519 165 165 165 307 307 307 F F F 26 520-539 168 168 168 312 312 312 F F F 27 540-559 168 168 168 312 312 312 F F F 28 560-579 168 168 168 311 311 311 F T T <--- 29 580-599 158 158 158 295 295 295 F F F Max gen: 172, Max flush: 325 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 172 172 172 322 322 322 F F F 1 020-039 172 172 172 325 325 325 F F F 2 040-059 172 172 172 325 325 325 F F F 3 060-079 172 172 172 325 325 325 F F F 4 080-099 165 165 165 307 307 307 F F F 5 100-119 173 173 173 326 326 326 F F F 6 120-139 173 173 173 326 326 326 F F F 7 140-159 172 172 172 324 324 324 F F F 8 160-179 173 173 173 326 326 326 F F F 9 180-199 166 166 166 308 308 308 F F F 10 200-219 167 167 167 310 310 310 F F F 11 220-239 167 167 167 310 310 310 F F F 12 240-259 173 173 173 326 326 326 F F F 13 260-279 173 173 173 326 326 326 F F F 14 280-299 171 171 171 320 320 320 F F F 15 300-319 172 172 172 323 323 323 F F F 16 320-339 173 173 173 326 326 326 F F F 17 340-359 170 170 170 316 316 316 F F F 18 360-379 170 170 170 318 318 318 F F F 19 380-399 166 166 166 309 309 309 F F F 20 400-419 168 168 168 311 311 311 F F F 21 420-439 173 173 173 326 326 326 F F F 22 440-459 170 170 170 318 318 318 F F F 23 460-479 169 169 169 315 315 315 F F F 24 480-499 173 173 173 326 326 326 F F F 25 500-519 165 165 165 307 307 307 F F F 26 520-539 168 168 168 312 312 312 F F F 27 540-559 168 168 168 312 312 312 F F F 28 560-579 173 173 173 326 326 326 F F F 29 580-599 158 158 158 295 295 295 F F F Max gen: 173, Max flush: 326 Begin loop 87 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.211518106Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.223272499Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.223593487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.223631823Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.223643896Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.223664613Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901} {"msg":"Crucible eed219ab-3f7e-470a-aff6-79d93d3b4d08 has session id: ca7574b7-a97b-43fe-95ff-43ccc5220f81","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.223993406Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Upstairs opts: Upstairs UUID: eed219ab-3f7e-470a-aff6-79d93d3b4d08, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224027904Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Crucible stats registered with UUID: eed219ab-3f7e-470a-aff6-79d93d3b4d08","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224042605Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:20.224055687Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224236783Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"io task","client":"1","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224271911Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"io task","client":"0","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224287511Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"io task","client":"2","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.22438504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"task":"crutest"} {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224399311Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901} {"msg":"eed219ab-3f7e-470a-aff6-79d93d3b4d08 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224421667Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224434529Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"client":"0","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.22444841Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"client":"1","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224462421Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"client":"2","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224475823Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"io task","client":"2","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224492392Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"io task","client":"0","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224506594Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"io task","client":"1","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"ds_connection connected from Ok(127.0.0.1:50539)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224681564Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"io task","client":"2","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"ds_connection connected from Ok(127.0.0.1:48745)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224708387Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"io task","client":"0","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"ds_connection connected from Ok(127.0.0.1:58743)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.224724477Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"io task","client":"1","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225269644Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"client":"0","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225298226Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"client":"1","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225413434Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"client":"2","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225649126Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225685713Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"[0]R flush_numbers[0..12]: [322, 325, 325, 325, 307, 326, 326, 324, 326, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225778694Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"[0]R generation[0..12]: [172, 172, 172, 172, 165, 173, 173, 172, 173, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225795933Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225812503Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"[1]R flush_numbers[0..12]: [322, 325, 325, 325, 307, 326, 326, 324, 326, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225827943Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"[1]R generation[0..12]: [172, 172, 172, 172, 165, 173, 173, 172, 173, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225846691Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225863081Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"[2]R flush_numbers[0..12]: [322, 325, 325, 325, 307, 326, 326, 324, 326, 308, 310, 310]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225878921Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"[2]R generation[0..12]: [172, 172, 172, 172, 165, 173, 173, 172, 173, 166, 167, 167]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225894232Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225910601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Max found gen is 174","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2026-03-14T05:07:20.225929409Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.22594372Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Next flush: 327","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225957252Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225970334Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"":"downstairs","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.225984265Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.226009969Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"client":"0","":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.226027708Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"client":"1","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.226041929Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"client":"2","":"downstairs","session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.22605627Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.226069701Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"eed219ab-3f7e-470a-aff6-79d93d3b4d08 is now active with session: ca7574b7-a97b-43fe-95ff-43ccc5220f81","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.226086521Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.226099003Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901,"session_id":"ca7574b7-a97b-43fe-95ff-43ccc5220f81"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.226112314Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901} {"msg":"Activated sub_volume eed219ab-3f7e-470a-aff6-79d93d3b4d08","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:20.226123877Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 403 len 16384 data: 17 14 17 16 02/30 Read block 247 len 16384 03/30 Write block 547 len 8192 data: 15 15 04/30 Flush 05/30 Write block 564 len 4096 data: 7 06/30 Write block 231 len 32768 data: 13 16 16 18 22 21 18 22 07/30 Write block 345 len 4096 data: 14 08/30 Read block 216 len 4096 09/30 Flush 10/30 Write block 126 len 32768 data: 10 8 9 7 11 12 12 15 11/30 Read block 32 len 40960 12/30 Write block 132 len 20480 data: 13 16 15 19 18 13/30 Read block 233 len 8192 14/30 Write block 196 len 12288 data: 12 14 11 15/30 Write block 486 len 16384 data: 11 15 15 11 16/30 Read block 156 len 8192 17/30 Read block 142 len 4096 18/30 Read block 101 len 8192 19/30 Write block 364 len 32768 data: 10 11 15 19 19 17 17 16 20/30 Write block 454 len 32768 data: 14 16 17 15 17 18 16 15 21/30 Flush 22/30 Write block 591 len 16384 data: 8 11 11 7 23/30 Write block 251 len 8192 data: 23 19 24/30 Read block 386 len 20480 25/30 Write block 288 len 8192 data: 12 11 26/30 Read block 266 len 40960 27/30 Read block 36 len 24576 28/30 Write block 472 len 32768 data: 7 6 9 7 12 15 16 16 29/30 Write block 28 len 4096 data: 14 30/30 Read block 323 len 4096 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 1 Done Done Sent false 1005 Acked Write 8 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 8 Done Done Sent false 1010 Acked Read 10 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Read 2 Done Done Sent false 1013 Acked Write 3 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Write 8 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Read 10 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Write 1 Done Done Sent false 1029 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 27 27 Done 30 30 3 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:20.237567256Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1901} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 172 172 172 322 322 322 F F F 1 020-039 172 172 172 325 325 325 T T F <--- 2 040-059 172 172 172 325 325 325 F F F 3 060-079 172 172 172 325 325 325 F F F 4 080-099 165 165 165 307 307 307 F F F 5 100-119 173 173 173 326 326 326 F F F 6 120-139 174 174 173 329 329 326 F F F <--- 7 140-159 172 172 172 324 324 324 F F F 8 160-179 173 173 173 326 326 326 F F F 9 180-199 174 174 166 329 329 308 F F F <--- 10 200-219 167 167 167 310 310 310 F F F 11 220-239 174 174 167 328 328 310 F F F <--- 12 240-259 173 173 173 326 326 326 T T F <--- 13 260-279 173 173 173 326 326 326 F F F 14 280-299 171 171 171 320 320 320 T T F <--- 15 300-319 172 172 172 323 323 323 F F F 16 320-339 173 173 173 326 326 326 F F F 17 340-359 174 174 170 328 328 316 F F F <--- 18 360-379 174 174 170 329 329 318 F F F <--- 19 380-399 166 166 166 309 309 309 F F F 20 400-419 174 174 168 327 327 311 F F T <--- 21 420-439 173 173 173 326 326 326 F F F 22 440-459 174 174 170 329 329 318 F F F <--- 23 460-479 174 174 169 329 329 315 T T F <--- 24 480-499 174 174 173 329 329 326 F F F <--- 25 500-519 165 165 165 307 307 307 F F F 26 520-539 168 168 168 312 312 312 F F F 27 540-559 174 174 168 327 327 312 F F T <--- 28 560-579 174 174 173 328 328 326 F F F <--- 29 580-599 158 158 158 295 295 295 T T F <--- Max gen: 174, Max flush: 329 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 172 172 172 322 322 322 F F F 1 020-039 175 175 175 330 330 330 F F F 2 040-059 172 172 172 325 325 325 F F F 3 060-079 172 172 172 325 325 325 F F F 4 080-099 165 165 165 307 307 307 F F F 5 100-119 173 173 173 326 326 326 F F F 6 120-139 174 174 174 329 329 329 F F F 7 140-159 172 172 172 324 324 324 F F F 8 160-179 173 173 173 326 326 326 F F F 9 180-199 174 174 174 329 329 329 F F F 10 200-219 167 167 167 310 310 310 F F F 11 220-239 174 174 174 328 328 328 F F F 12 240-259 175 175 175 330 330 330 F F F 13 260-279 173 173 173 326 326 326 F F F 14 280-299 175 175 175 330 330 330 F F F 15 300-319 172 172 172 323 323 323 F F F 16 320-339 173 173 173 326 326 326 F F F 17 340-359 174 174 174 328 328 328 F F F 18 360-379 174 174 174 329 329 329 F F F 19 380-399 166 166 166 309 309 309 F F F 20 400-419 174 174 174 327 327 327 F F F 21 420-439 173 173 173 326 326 326 F F F 22 440-459 174 174 174 329 329 329 F F F 23 460-479 175 175 175 330 330 330 F F F 24 480-499 174 174 174 329 329 329 F F F 25 500-519 165 165 165 307 307 307 F F F 26 520-539 168 168 168 312 312 312 F F F 27 540-559 174 174 174 327 327 327 F F F 28 560-579 174 174 174 328 328 328 F F F 29 580-599 175 175 175 330 330 330 F F F Max gen: 175, Max flush: 330 Begin loop 88 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.566407282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:24.577033204Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.577084282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.587261047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.58755883Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.587597765Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.587609778Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.587627686Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909} {"msg":"Crucible ca3c64de-ea24-437c-8932-04def2318da6 has session id: 8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.587981934Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Upstairs opts: Upstairs UUID: ca3c64de-ea24-437c-8932-04def2318da6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588027835Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Crucible stats registered with UUID: ca3c64de-ea24-437c-8932-04def2318da6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588043855Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:24.588060085Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588209761Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"io task","client":"1","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588245458Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"io task","client":"0","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588261438Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"io task","client":"2","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588362525Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"task":"crutest"} {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588379114Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909} {"msg":"ca3c64de-ea24-437c-8932-04def2318da6 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588400201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588413033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"client":"0","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588427044Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"client":"1","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588441045Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"client":"2","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588454996Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"io task","client":"2","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588469237Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"io task","client":"0","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588483179Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"io task","client":"1","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"ds_connection connected from Ok(127.0.0.1:58339)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588545779Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"io task","client":"1","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"ds_connection connected from Ok(127.0.0.1:56224)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.588561769Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"io task","client":"2","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"ds_connection connected from Ok(127.0.0.1:50494)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.58857661Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"io task","client":"0","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589331256Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"client":"1","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589358818Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"client":"0","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589376028Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"client":"2","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589710308Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589758907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"[0]R flush_numbers[0..12]: [322, 330, 325, 325, 307, 326, 329, 324, 326, 329, 310, 328]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589832661Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"[0]R generation[0..12]: [172, 175, 172, 172, 165, 173, 174, 172, 173, 174, 167, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.58985021Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589866759Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"[1]R flush_numbers[0..12]: [322, 330, 325, 325, 307, 326, 329, 324, 326, 329, 310, 328]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589882389Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"[1]R generation[0..12]: [172, 175, 172, 172, 165, 173, 174, 172, 173, 174, 167, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589898279Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589914759Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"[2]R flush_numbers[0..12]: [322, 330, 325, 325, 307, 326, 329, 324, 326, 329, 310, 328]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589930599Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"[2]R generation[0..12]: [172, 175, 172, 172, 165, 173, 174, 172, 173, 174, 167, 174]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589946499Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589962889Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 176","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589985035Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.589999706Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Next flush: 331","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590012957Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590026469Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"":"downstairs","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.5900408Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590053762Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"client":"0","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590067923Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"client":"1","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590081394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"client":"2","":"downstairs","session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590095785Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590112225Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"ca3c64de-ea24-437c-8932-04def2318da6 is now active with session: 8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590127395Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590140227Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909,"session_id":"8ab2f85b-9c9b-4393-b6ac-e7b1cbae4b8e"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590153719Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909} {"msg":"Activated sub_volume ca3c64de-ea24-437c-8932-04def2318da6","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:24.590165262Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 0 len 4096 data: 5 02/30 Flush 03/30 Write block 203 len 24576 data: 16 14 15 12 12 17 04/30 Write block 113 len 4096 data: 10 05/30 Read block 386 len 12288 06/30 Write block 67 len 4096 data: 13 07/30 Read block 9 len 16384 08/30 Read block 160 len 8192 09/30 Read block 550 len 12288 10/30 Read block 468 len 8192 11/30 Read block 538 len 32768 12/30 Write block 378 len 4096 data: 13 13/30 Read block 86 len 12288 14/30 Read block 301 len 4096 15/30 Write block 530 len 36864 data: 15 17 16 15 14 11 10 10 11 16/30 Write block 84 len 20480 data: 12 15 16 14 11 17/30 Write block 192 len 40960 data: 14 13 15 17 13 15 12 12 12 12 18/30 Flush 19/30 Write block 229 len 40960 data: 9 12 14 17 17 19 23 22 19 23 20/30 Write block 520 len 36864 data: 12 10 12 14 12 14 15 16 15 21/30 Read block 176 len 32768 22/30 Read block 582 len 16384 23/30 Write block 76 len 4096 data: 15 24/30 Write block 479 len 8192 data: 17 15 25/30 Read block 53 len 12288 26/30 Write block 352 len 12288 data: 16 13 14 27/30 Read block 211 len 20480 28/30 Write block 553 len 16384 data: 12 13 12 13 29/30 Read block 323 len 8192 30/30 Write block 332 len 28672 data: 11 12 14 16 17 14 16 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 6 Sent Done Done false 1003 Acked Write 1 Sent Done Done false 1004 Acked Read 3 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Read 8 Sent Done Done false 1011 Acked Write 1 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:24.601383762Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1909} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 172 176 176 322 331 331 T F F <--- 1 020-039 175 175 175 330 330 330 F F F 2 040-059 172 172 172 325 325 325 F F F 3 060-079 172 176 176 325 332 332 F T T <--- 4 080-099 165 176 176 307 332 332 F F F <--- 5 100-119 173 176 176 326 332 332 F F F <--- 6 120-139 174 174 174 329 329 329 F F F 7 140-159 172 172 172 324 324 324 F F F 8 160-179 173 173 173 326 326 326 F F F 9 180-199 174 176 176 329 332 332 F F F <--- 10 200-219 167 176 176 310 332 332 F F F <--- 11 220-239 174 174 174 328 328 328 F T T <--- 12 240-259 175 175 175 330 330 330 F F F 13 260-279 173 173 173 326 326 326 F F F 14 280-299 175 175 175 330 330 330 F F F 15 300-319 172 172 172 323 323 323 F F F 16 320-339 173 173 173 326 326 326 F T T <--- 17 340-359 174 174 174 328 328 328 F T T <--- 18 360-379 174 176 176 329 332 332 F F F <--- 19 380-399 166 166 166 309 309 309 F F F 20 400-419 174 174 174 327 327 327 F F F 21 420-439 173 173 173 326 326 326 F F F 22 440-459 174 174 174 329 329 329 F F F 23 460-479 175 175 175 330 330 330 F T T <--- 24 480-499 174 174 174 329 329 329 F T T <--- 25 500-519 165 165 165 307 307 307 F F F 26 520-539 168 176 176 312 332 332 F T T <--- 27 540-559 174 174 174 327 327 327 F T T <--- 28 560-579 174 174 174 328 328 328 F F F 29 580-599 175 175 175 330 330 330 F F F Max gen: 176, Max flush: 332 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 331 331 331 F F F 1 020-039 175 175 175 330 330 330 F F F 2 040-059 172 172 172 325 325 325 F F F 3 060-079 177 177 177 333 333 333 F F F 4 080-099 176 176 176 332 332 332 F F F 5 100-119 176 176 176 332 332 332 F F F 6 120-139 174 174 174 329 329 329 F F F 7 140-159 172 172 172 324 324 324 F F F 8 160-179 173 173 173 326 326 326 F F F 9 180-199 176 176 176 332 332 332 F F F 10 200-219 176 176 176 332 332 332 F F F 11 220-239 177 177 177 333 333 333 F F F 12 240-259 175 175 175 330 330 330 F F F 13 260-279 173 173 173 326 326 326 F F F 14 280-299 175 175 175 330 330 330 F F F 15 300-319 172 172 172 323 323 323 F F F 16 320-339 177 177 177 333 333 333 F F F 17 340-359 177 177 177 333 333 333 F F F 18 360-379 176 176 176 332 332 332 F F F 19 380-399 166 166 166 309 309 309 F F F 20 400-419 174 174 174 327 327 327 F F F 21 420-439 173 173 173 326 326 326 F F F 22 440-459 174 174 174 329 329 329 F F F 23 460-479 177 177 177 333 333 333 F F F 24 480-499 177 177 177 333 333 333 F F F 25 500-519 165 165 165 307 307 307 F F F 26 520-539 177 177 177 333 333 333 F F F 27 540-559 177 177 177 333 333 333 F F F 28 560-579 174 174 174 328 328 328 F F F 29 580-599 175 175 175 330 330 330 F F F Max gen: 177, Max flush: 333 Begin loop 89 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.937396839Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:28.94799456Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.94804186Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.958199307Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.958493492Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.958530179Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.958542601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.95856049Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917} {"msg":"Crucible 63fd4810-8ef1-4428-9ba3-9f2403afa3ad has session id: 5f4f24d0-ab5a-4286-96df-af11780ca73e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.958885655Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Upstairs opts: Upstairs UUID: 63fd4810-8ef1-4428-9ba3-9f2403afa3ad, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.958911519Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Crucible stats registered with UUID: 63fd4810-8ef1-4428-9ba3-9f2403afa3ad","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.95892615Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:28.958939112Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959119898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"io task","client":"0","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959156355Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"io task","client":"1","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959172015Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"io task","client":"2","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959264178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"task":"crutest"} {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959281247Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917} {"msg":"63fd4810-8ef1-4428-9ba3-9f2403afa3ad active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959293679Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959306431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"client":"0","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959320492Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"client":"1","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959334983Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"client":"2","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959348994Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"io task","client":"2","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959384722Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"io task","client":"0","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959409746Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"io task","client":"1","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"ds_connection connected from Ok(127.0.0.1:39872)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959425816Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"io task","client":"2","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"ds_connection connected from Ok(127.0.0.1:45030)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959441076Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"io task","client":"0","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"ds_connection connected from Ok(127.0.0.1:47221)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.959542033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"io task","client":"1","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960179922Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"client":"1","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960208034Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"client":"0","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960257263Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"client":"2","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960536748Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960579271Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"[0]R flush_numbers[0..12]: [331, 330, 325, 333, 332, 332, 329, 324, 326, 332, 332, 333]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960657532Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"[0]R generation[0..12]: [176, 175, 172, 177, 176, 176, 174, 172, 173, 176, 176, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960674991Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.96069212Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"[1]R flush_numbers[0..12]: [331, 330, 325, 333, 332, 332, 329, 324, 326, 332, 332, 333]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.96070813Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"[1]R generation[0..12]: [176, 175, 172, 177, 176, 176, 174, 172, 173, 176, 176, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.96072389Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.9607403Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"[2]R flush_numbers[0..12]: [331, 330, 325, 333, 332, 332, 329, 324, 326, 332, 332, 333]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.96075616Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"[2]R generation[0..12]: [176, 175, 172, 177, 176, 176, 174, 172, 173, 176, 176, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.96077181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.96078826Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960801601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:07:28.960815683Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Next flush: 334","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960833012Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960846353Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"":"downstairs","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960860824Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960875335Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"client":"0","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960889766Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"client":"1","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960903837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"client":"2","":"downstairs","session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960918428Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960934248Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"63fd4810-8ef1-4428-9ba3-9f2403afa3ad is now active with session: 5f4f24d0-ab5a-4286-96df-af11780ca73e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960949718Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.96096238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917,"session_id":"5f4f24d0-ab5a-4286-96df-af11780ca73e"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960975752Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917} {"msg":"Activated sub_volume 63fd4810-8ef1-4428-9ba3-9f2403afa3ad","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:28.960987165Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 243 len 32768 data: 13 16 19 21 24 29 28 28 02/30 Write block 315 len 28672 data: 13 10 9 7 9 11 12 03/30 Read block 343 len 8192 04/30 Read block 451 len 8192 05/30 Read block 243 len 12288 06/30 Write block 86 len 4096 data: 17 07/30 Read block 175 len 24576 08/30 Read block 309 len 20480 09/30 Write block 419 len 20480 data: 13 13 16 14 15 10/30 Read block 578 len 28672 11/30 Write block 153 len 12288 data: 10 12 13 12/30 Write block 101 len 8192 data: 13 15 13/30 Read block 4 len 4096 14/30 Read block 246 len 16384 15/30 Write block 73 len 20480 data: 13 17 15 16 13 16/30 Read block 438 len 8192 17/30 Read block 543 len 40960 18/30 Write block 472 len 12288 data: 8 7 10 19/30 Write block 484 len 28672 data: 15 14 12 16 16 12 12 20/30 Flush 21/30 Write block 443 len 32768 data: 17 14 13 15 16 13 15 12 22/30 Read block 489 len 32768 23/30 Write block 188 len 20480 data: 16 15 16 15 15 24/30 Write block 87 len 20480 data: 15 12 9 12 13 25/30 Write block 338 len 20480 data: 17 14 13 15 17 26/30 Read block 111 len 40960 27/30 Read block 307 len 16384 28/30 Write block 383 len 16384 data: 10 9 7 10 29/30 Read block 260 len 8192 30/30 Write block 369 len 16384 data: 18 18 17 13 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Read 2 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Read 6 Sent Done Done false 1007 Acked Read 5 Sent Done Done false 1008 Acked Write 5 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Read 1 Sent Done Done false 1013 Acked Read 4 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Read 10 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Write 5 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 1 1 27 Done 5 29 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:28.972481551Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1917} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 331 331 331 F F F 1 020-039 175 175 175 330 330 330 F F F 2 040-059 172 172 172 325 325 325 F F F 3 060-079 177 178 178 333 334 334 F F F <--- 4 080-099 176 178 178 332 334 334 F T T <--- 5 100-119 176 178 178 332 334 334 F F F <--- 6 120-139 174 174 174 329 329 329 F F F 7 140-159 172 178 178 324 334 334 F F F <--- 8 160-179 173 173 173 326 326 326 F F F 9 180-199 176 176 176 332 332 332 F T T <--- 10 200-219 176 176 176 332 332 332 F F F 11 220-239 177 177 177 333 333 333 F F F 12 240-259 175 178 178 330 334 334 T F F <--- 13 260-279 173 173 173 326 326 326 F F F 14 280-299 175 175 175 330 330 330 F F F 15 300-319 172 178 178 323 334 334 T F F <--- 16 320-339 177 178 178 333 334 334 T T T <--- 17 340-359 177 177 177 333 333 333 F T T <--- 18 360-379 176 176 176 332 332 332 F T T <--- 19 380-399 166 166 166 309 309 309 F T T <--- 20 400-419 174 178 178 327 334 334 F F F <--- 21 420-439 173 178 178 326 334 334 F F F <--- 22 440-459 174 174 174 329 329 329 F T T <--- 23 460-479 177 178 178 333 334 334 F F F <--- 24 480-499 177 178 178 333 334 334 F F F <--- 25 500-519 165 165 165 307 307 307 F F F 26 520-539 177 177 177 333 333 333 F F F 27 540-559 177 177 177 333 333 333 F F F 28 560-579 174 174 174 328 328 328 F F F 29 580-599 175 175 175 330 330 330 F F F Max gen: 178, Max flush: 334 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 331 331 331 F F F 1 020-039 175 175 175 330 330 330 F F F 2 040-059 172 172 172 325 325 325 F F F 3 060-079 178 178 178 334 334 334 F F F 4 080-099 179 179 179 335 335 335 F F F 5 100-119 178 178 178 334 334 334 F F F 6 120-139 174 174 174 329 329 329 F F F 7 140-159 178 178 178 334 334 334 F F F 8 160-179 173 173 173 326 326 326 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 176 176 176 332 332 332 F F F 11 220-239 177 177 177 333 333 333 F F F 12 240-259 178 178 178 334 334 334 F F F 13 260-279 173 173 173 326 326 326 F F F 14 280-299 175 175 175 330 330 330 F F F 15 300-319 178 178 178 334 334 334 F F F 16 320-339 179 179 179 335 335 335 F F F 17 340-359 179 179 179 335 335 335 F F F 18 360-379 179 179 179 335 335 335 F F F 19 380-399 179 179 179 335 335 335 F F F 20 400-419 178 178 178 334 334 334 F F F 21 420-439 178 178 178 334 334 334 F F F 22 440-459 179 179 179 335 335 335 F F F 23 460-479 178 178 178 334 334 334 F F F 24 480-499 178 178 178 334 334 334 F F F 25 500-519 165 165 165 307 307 307 F F F 26 520-539 177 177 177 333 333 333 F F F 27 540-559 177 177 177 333 333 333 F F F 28 560-579 174 174 174 328 328 328 F F F 29 580-599 175 175 175 330 330 330 F F F Max gen: 179, Max flush: 335 Begin loop 90 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.313310594Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:33.323912731Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.32396184Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.334166167Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.334466938Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.334503346Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.334519036Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.334536994Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925} {"msg":"Crucible 575dd443-ad13-45ae-b802-898eb44a059d has session id: 65334d18-fd4a-43ef-9440-d271e8392679","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.334883377Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Upstairs opts: Upstairs UUID: 575dd443-ad13-45ae-b802-898eb44a059d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.334918155Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Crucible stats registered with UUID: 575dd443-ad13-45ae-b802-898eb44a059d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.334934225Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:33.334947966Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335105197Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"io task","client":"2","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335140065Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"io task","client":"0","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335156005Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"io task","client":"1","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335245419Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"task":"crutest"} {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.33525941Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925} {"msg":"575dd443-ad13-45ae-b802-898eb44a059d active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335289771Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335303663Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"client":"0","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335317834Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"client":"1","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335331645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"client":"2","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335374388Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"io task","client":"2","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335402361Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"io task","client":"1","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335417831Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"io task","client":"0","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"ds_connection connected from Ok(127.0.0.1:33684)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335433141Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"io task","client":"2","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"ds_connection connected from Ok(127.0.0.1:40404)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335448062Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"io task","client":"1","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"ds_connection connected from Ok(127.0.0.1:59629)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.335462503Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"io task","client":"0","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.33621438Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"client":"2","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336242942Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"client":"0","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336267977Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"client":"1","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336594571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336608802Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"[0]R flush_numbers[0..12]: [331, 330, 325, 334, 335, 334, 329, 334, 326, 335, 332, 333]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336649397Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"[0]R generation[0..12]: [176, 175, 172, 178, 179, 178, 174, 178, 173, 179, 176, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336666496Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336683406Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"[1]R flush_numbers[0..12]: [331, 330, 325, 334, 335, 334, 329, 334, 326, 335, 332, 333]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336698976Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"[1]R generation[0..12]: [176, 175, 172, 178, 179, 178, 174, 178, 173, 179, 176, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336717064Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336733504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"[2]R flush_numbers[0..12]: [331, 330, 325, 334, 335, 334, 329, 334, 326, 335, 332, 333]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336749414Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"[2]R generation[0..12]: [176, 175, 172, 178, 179, 178, 174, 178, 173, 179, 176, 177]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336764984Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336781654Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Max found gen is 180","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336795435Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Generation requested: 180 >= found:180","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336809586Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Next flush: 336","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336822568Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"Wait for a query_work_queue command to finish before sending IO } {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336841946Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"":"downstairs","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336856057Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336868559Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"client":"0","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.33688247Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"client":"1","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336896652Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"client":"2","":"downstairs","session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336919547Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336932969Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"575dd443-ad13-45ae-b802-898eb44a059d is now active with session: 65334d18-fd4a-43ef-9440-d271e8392679","v":0,"name":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 crucible","level":30,"time":"2026-03-14T05:07:33.336952157Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336965089Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925,"session_id":"65334d18-fd4a-43ef-9440-d271e8392679"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.33697857Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925} {"msg":"Activated sub_volume 575dd443-ad13-45ae-b802-898eb44a059d","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:33.336990393Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 295 len 32768 data: 13 12 10 10 10 10 9 9 02/30 Write block 238 len 8192 data: 24 21 03/30 Read block 66 len 36864 04/30 Flush 05/30 Write block 366 len 28672 data: 16 20 20 19 19 18 14 06/30 Read block 62 len 12288 07/30 Write block 290 len 36864 data: 12 17 15 13 13 14 13 11 11 08/30 Write block 590 len 36864 data: 10 9 12 12 8 6 6 5 5 09/30 Write block 354 len 36864 data: 15 15 16 13 13 11 13 13 14 10/30 Flush 11/30 Write block 332 len 12288 data: 12 13 15 12/30 Write block 496 len 32768 data: 14 12 12 14 13 15 15 15 13/30 Write block 155 len 12288 data: 14 14 16 14/30 Write block 422 len 40960 data: 15 16 14 13 10 11 14 16 10 11 15/30 Read block 390 len 28672 16/30 Read block 311 len 4096 17/30 Read block 455 len 24576 18/30 Flush 19/30 Write block 567 len 4096 data: 12 20/30 Read block 180 len 36864 21/30 Read block 266 len 24576 22/30 Flush 23/30 Write block 368 len 32768 data: 21 20 20 19 15 12 11 13 24/30 Read block 53 len 36864 25/30 Write block 35 len 4096 data: 16 26/30 Read block 585 len 4096 27/30 Write block 319 len 24576 data: 10 12 13 10 10 10 28/30 Read block 309 len 16384 29/30 Write block 494 len 40960 data: 17 19 15 13 13 15 14 16 16 16 30/30 Write block 101 len 20480 data: 14 16 17 13 13 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Write 2 Sent Done Done false 1002 Acked Read 9 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Write 9 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 7 Sent Done Done false 1015 Acked Read 1 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Read 9 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:33.349601018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1925} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 331 331 331 F F F 1 020-039 175 175 175 330 330 330 F T T <--- 2 040-059 172 172 172 325 325 325 F F F 3 060-079 178 178 178 334 334 334 F F F 4 080-099 179 179 179 335 335 335 F F F 5 100-119 178 178 178 334 334 334 F T T <--- 6 120-139 174 174 174 329 329 329 F F F 7 140-159 178 180 180 334 338 338 F F F <--- 8 160-179 173 173 173 326 326 326 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 176 176 176 332 332 332 F F F 11 220-239 177 180 180 333 336 336 F F F <--- 12 240-259 178 178 178 334 334 334 F F F 13 260-279 173 173 173 326 326 326 F F F 14 280-299 175 180 180 330 337 337 F F F <--- 15 300-319 178 180 180 334 336 336 F T T <--- 16 320-339 179 180 180 335 338 338 F T T <--- 17 340-359 179 180 180 335 337 337 F F F <--- 18 360-379 179 180 180 335 337 337 F T T <--- 19 380-399 179 179 179 335 335 335 F F F 20 400-419 178 178 178 334 334 334 F F F 21 420-439 178 180 180 334 338 338 F F F <--- 22 440-459 179 179 179 335 335 335 F F F 23 460-479 178 178 178 334 334 334 F F F 24 480-499 178 180 180 334 338 338 F T T <--- 25 500-519 165 180 180 307 338 338 F T T <--- 26 520-539 177 177 177 333 333 333 F F F 27 540-559 177 177 177 333 333 333 F F F 28 560-579 174 180 180 328 339 339 F F F <--- 29 580-599 175 180 180 330 337 337 F F F <--- Max gen: 180, Max flush: 339 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 331 331 331 F F F 1 020-039 181 181 181 340 340 340 F F F 2 040-059 172 172 172 325 325 325 F F F 3 060-079 178 178 178 334 334 334 F F F 4 080-099 179 179 179 335 335 335 F F F 5 100-119 181 181 181 340 340 340 F F F 6 120-139 174 174 174 329 329 329 F F F 7 140-159 180 180 180 338 338 338 F F F 8 160-179 173 173 173 326 326 326 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 176 176 176 332 332 332 F F F 11 220-239 180 180 180 336 336 336 F F F 12 240-259 178 178 178 334 334 334 F F F 13 260-279 173 173 173 326 326 326 F F F 14 280-299 180 180 180 337 337 337 F F F 15 300-319 181 181 181 340 340 340 F F F 16 320-339 181 181 181 340 340 340 F F F 17 340-359 180 180 180 337 337 337 F F F 18 360-379 181 181 181 340 340 340 F F F 19 380-399 179 179 179 335 335 335 F F F 20 400-419 178 178 178 334 334 334 F F F 21 420-439 180 180 180 338 338 338 F F F 22 440-459 179 179 179 335 335 335 F F F 23 460-479 178 178 178 334 334 334 F F F 24 480-499 181 181 181 340 340 340 F F F 25 500-519 181 181 181 340 340 340 F F F 26 520-539 177 177 177 333 333 333 F F F 27 540-559 177 177 177 333 333 333 F F F 28 560-579 180 180 180 339 339 339 F F F 29 580-599 180 180 180 337 337 337 F F F Max gen: 181, Max flush: 340 Begin loop 91 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.689329483Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.700955487Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.701271848Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.701310743Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.701323276Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.701341084Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933} {"msg":"Crucible 1943a716-b95c-4e37-a547-b88ffea93ea0 has session id: b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.70165113Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Upstairs opts: Upstairs UUID: 1943a716-b95c-4e37-a547-b88ffea93ea0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.701685398Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Crucible stats registered with UUID: 1943a716-b95c-4e37-a547-b88ffea93ea0","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.701701448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:37.701715199Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.7018733Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"io task","client":"2","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.701907438Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"io task","client":"0","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.701923269Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"io task","client":"1","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.70203211Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"task":"crutest"} {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.70204888Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933} {"msg":"1943a716-b95c-4e37-a547-b88ffea93ea0 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.70206383Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702076602Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"client":"0","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702090743Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"client":"1","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702104784Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"client":"2","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702118436Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"io task","client":"2","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702132437Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"io task","client":"0","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702151445Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"io task","client":"1","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"ds_connection connected from Ok(127.0.0.1:32942)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702238301Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"io task","client":"0","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"ds_connection connected from Ok(127.0.0.1:33459)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702384039Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"io task","client":"2","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"ds_connection connected from Ok(127.0.0.1:45336)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702412601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"io task","client":"1","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702946006Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"client":"2","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.702989988Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"client":"1","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703007637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"client":"0","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703368191Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703415291Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"[0]R flush_numbers[0..12]: [331, 340, 325, 334, 335, 340, 329, 338, 326, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703450339Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"[0]R generation[0..12]: [176, 181, 172, 178, 179, 181, 174, 180, 173, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703468078Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703484777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"[1]R flush_numbers[0..12]: [331, 340, 325, 334, 335, 340, 329, 338, 326, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703500637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"[1]R generation[0..12]: [176, 181, 172, 178, 179, 181, 174, 180, 173, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703516277Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703532447Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"[2]R flush_numbers[0..12]: [331, 340, 325, 334, 335, 340, 329, 338, 326, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703548137Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"[2]R generation[0..12]: [176, 181, 172, 178, 179, 181, 174, 180, 173, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703563927Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703580247Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Max found gen is 182","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:07:37.703593409Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703615225Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Next flush: 341","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703628637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703642318Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"":"downstairs","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703656479Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703668961Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"client":"0","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703682883Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"client":"1","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703696904Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"client":"2","":"downstairs","session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"All required reconciliation work is completed","v":0,"name":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 crucible","level":30,"time":"2026-03-14T05:07:37.703713913Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703727515Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"1943a716-b95c-4e37-a547-b88ffea93ea0 is now active with session: b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703742445Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703755047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933,"session_id":"b1d7e6ac-3de7-48bc-aaf9-dd3a3cf41dea"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703768199Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933} {"msg":"Activated sub_volume 1943a716-b95c-4e37-a547-b88ffea93ea0","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:37.703779532Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 548 len 20480 data: 16 15 17 14 13 02/30 Write block 395 len 4096 data: 16 03/30 Write block 249 len 40960 data: 29 29 24 20 16 16 16 15 13 13 04/30 Read block 384 len 24576 05/30 Flush 06/30 Write block 376 len 12288 data: 13 11 14 07/30 Read block 515 len 8192 08/30 Read block 138 len 16384 09/30 Read block 174 len 32768 10/30 Read block 149 len 28672 11/30 Read block 377 len 32768 12/30 Read block 524 len 24576 13/30 Write block 105 len 12288 data: 14 14 10 14/30 Write block 578 len 32768 data: 15 15 17 17 18 20 19 14 15/30 Write block 19 len 20480 data: 12 12 13 13 10 16/30 Read block 512 len 8192 17/30 Read block 96 len 8192 18/30 Read block 30 len 28672 19/30 Read block 317 len 36864 20/30 Write block 395 len 28672 data: 17 16 14 14 15 14 20 21/30 Read block 390 len 36864 22/30 Read block 101 len 28672 23/30 Read block 477 len 20480 24/30 Read block 572 len 12288 25/30 Flush 26/30 Write block 131 len 32768 data: 13 14 17 16 20 19 17 16 27/30 Read block 328 len 36864 28/30 Write block 100 len 12288 data: 11 15 17 29/30 Read block 323 len 8192 30/30 Read block 365 len 20480 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Write 10 Done Sent Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 3 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Read 7 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 9 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 1 29 Done 30 2 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:37.716280516Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1933} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 176 182 342 331 342 F F F <--- 1 020-039 182 181 182 342 340 342 F F F <--- 2 040-059 172 172 172 325 325 325 F F F 3 060-079 178 178 178 334 334 334 F F F 4 080-099 179 179 179 335 335 335 F F F 5 100-119 182 181 182 342 340 342 T F T <--- 6 120-139 174 174 174 329 329 329 T F T <--- 7 140-159 180 180 180 338 338 338 F F F 8 160-179 173 173 173 326 326 326 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 176 176 176 332 332 332 F F F 11 220-239 180 180 180 336 336 336 F F F 12 240-259 182 178 182 341 334 341 F F F <--- 13 260-279 173 173 173 326 326 326 F F F 14 280-299 180 180 180 337 337 337 F F F 15 300-319 181 181 181 340 340 340 F F F 16 320-339 181 181 181 340 340 340 F F F 17 340-359 180 180 180 337 337 337 F F F 18 360-379 182 181 182 342 340 342 F F F <--- 19 380-399 182 179 182 342 335 342 F T F <--- 20 400-419 182 178 182 342 334 342 F F F <--- 21 420-439 180 180 180 338 338 338 F F F 22 440-459 179 179 179 335 335 335 F F F 23 460-479 178 178 178 334 334 334 F F F 24 480-499 181 181 181 340 340 340 F F F 25 500-519 181 181 181 340 340 340 F F F 26 520-539 177 177 177 333 333 333 F F F 27 540-559 182 177 182 341 333 341 F T F <--- 28 560-579 182 180 182 342 339 342 F F F <--- 29 580-599 182 180 182 342 337 342 F F F <--- Max gen: 182, Max flush: 342 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 182 182 182 342 342 342 F F F 2 040-059 172 172 172 325 325 325 F F F 3 060-079 178 178 178 334 334 334 F F F 4 080-099 179 179 179 335 335 335 F F F 5 100-119 183 183 183 343 343 343 F F F 6 120-139 183 183 183 343 343 343 F F F 7 140-159 180 180 180 338 338 338 F F F 8 160-179 173 173 173 326 326 326 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 176 176 176 332 332 332 F F F 11 220-239 180 180 180 336 336 336 F F F 12 240-259 182 182 182 341 341 341 F F F 13 260-279 173 173 173 326 326 326 F F F 14 280-299 180 180 180 337 337 337 F F F 15 300-319 181 181 181 340 340 340 F F F 16 320-339 181 181 181 340 340 340 F F F 17 340-359 180 180 180 337 337 337 F F F 18 360-379 182 182 182 342 342 342 F F F 19 380-399 182 182 182 342 342 342 F F F 20 400-419 182 182 182 342 342 342 F F F 21 420-439 180 180 180 338 338 338 F F F 22 440-459 179 179 179 335 335 335 F F F 23 460-479 178 178 178 334 334 334 F F F 24 480-499 181 181 181 340 340 340 F F F 25 500-519 181 181 181 340 340 340 F F F 26 520-539 177 177 177 333 333 333 F F F 27 540-559 182 182 182 341 341 341 F F F 28 560-579 182 182 182 342 342 342 F F F 29 580-599 182 182 182 342 342 342 F F F Max gen: 183, Max flush: 343 Begin loop 92 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.036539134Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:42.047121724Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.047167515Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.057553508Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.057849732Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.057887848Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.057900031Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.05791786Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941} {"msg":"Crucible 988d2757-585e-4db0-ac4f-b9dc7f39f8f9 has session id: 5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058233251Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Upstairs opts: Upstairs UUID: 988d2757-585e-4db0-ac4f-b9dc7f39f8f9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058272397Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Crucible stats registered with UUID: 988d2757-585e-4db0-ac4f-b9dc7f39f8f9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058287327Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:42.058300609Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058455492Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"io task","client":"2","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058478637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"io task","client":"0","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058493178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"io task","client":"1","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058626534Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"task":"crutest"} {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058643074Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941} {"msg":"988d2757-585e-4db0-ac4f-b9dc7f39f8f9 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058666699Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058682049Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"client":"0","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058696021Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"client":"1","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058709832Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"client":"2","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058723563Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"io task","client":"2","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058737954Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"io task","client":"0","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058752155Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"io task","client":"1","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:57265)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058806111Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"io task","client":"2","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:48056)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058830006Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"io task","client":"0","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:63236)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.058869511Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"io task","client":"1","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.059593696Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"client":"2","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.059616912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"client":"1","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.059666371Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"client":"0","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.059989108Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060038907Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"[0]R flush_numbers[0..12]: [342, 342, 325, 334, 335, 343, 343, 338, 326, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060075394Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"[0]R generation[0..12]: [182, 182, 172, 178, 179, 183, 183, 180, 173, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060092533Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060111901Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"[1]R flush_numbers[0..12]: [342, 342, 325, 334, 335, 343, 343, 338, 326, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060127901Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"[1]R generation[0..12]: [182, 182, 172, 178, 179, 183, 183, 180, 173, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060143751Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.06016018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"[2]R flush_numbers[0..12]: [342, 342, 325, 334, 335, 343, 343, 338, 326, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.06017626Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"[2]R generation[0..12]: [182, 182, 172, 178, 179, 183, 183, 180, 173, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.06019204Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.06020844Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":Wait for a query_work_queue command to finish before sending IO 1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060228188Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060242419Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Next flush: 344","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.06025602Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060269762Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"":"downstairs","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060284752Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060297754Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"client":"0","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060311875Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"client":"1","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060328914Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"clientDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":"2","":"downstairs","session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060346813Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060360315Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"988d2757-585e-4db0-ac4f-b9dc7f39f8f9 is now active with session: 5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060375755Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060388347Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941,"session_id":"5b03bc53-bbfa-4b3b-861f-ccc979f0f3ad"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060401399Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941} {"msg":"Activated sub_volume 988d2757-585e-4db0-ac4f-b9dc7f39f8f9","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:42.060412712Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 167 len 36864 data: 12 11 11 10 6 8 9 12 13 02/30 Write block 258 len 8192 data: 14 13 03/30 Read block 131 len 20480 04/30 Write block 369 len 4096 data: 21 05/30 Read block 245 len 24576 06/30 Flush 07/30 Write block 399 len 28672 data: 16 15 21 21 18 15 18 08/30 Read block 473 len 16384 09/30 Flush 10/30 Write block 348 len 40960 data: 15 17 19 14 17 14 16 16 17 14 11/30 Write block 480 len 4096 data: 16 12/30 Write block 449 len 40960 data: 16 13 13 12 15 15 17 18 16 18 13/30 Flush 14/30 Write block 99 len 24576 data: 9 12 16 18 18 14 15/30 Read block 174 len 8192 16/30 Read block 277 len 16384 17/30 Write block 398 len 36864 data: 15 17 16 22 22 19 16 19 17 18/30 Read block 298 len 36864 19/30 Write block 571 len 4096 data: 11 20/30 Read block 48 len 28672 21/30 Read block 337 len 8192 22/30 Write block 107 len 40960 data: 11 10 11 10 9 10 11 9 8 10 23/30 Write block 155 len 12288 data: 15 15 17 24/30 Read block 253 len 4096 25/30 Read block 111 len 8192 26/30 Read block 129 len 16384 27/30 Write block 314 len 4096 data: 13 28/30 Write block 587 len 12288 data: 12 10 12 29/30 Read block 583 len 4096 30/30 Read block 176 len 40960 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Read 5 Sent Done Done false 1003 Acked Write 1 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 6 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Read 10 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 0 28 Done 2 30 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1012)) Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:42.072559898Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1941} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 182 182 182 342 342 342 F F F 2 040-059 172 172 172 325 325 325 F F F 3 060-079 178 178 178 334 334 334 F F F 4 080-099 179 179 179 335 335 335 F T T <--- 5 100-119 183 183 183 343 343 343 F T T <--- 6 120-139 183 183 183 343 343 343 F F F 7 140-159 180 180 180 338 338 338 F T T <--- 8 160-179 173 184 184 326 344 344 T F F <--- 9 180-199 179 179 179 335 335 335 F F F 10 200-219 176 176 176 332 332 332 F F F 11 220-239 180 180 180 336 336 336 F F F 12 240-259 182 184 184 341 344 344 T F F <--- 13 260-279 173 173 173 326 326 326 F F F 14 280-299 180 180 180 337 337 337 F F F 15 300-319 181 181 181 340 340 340 F T T <--- 16 320-339 181 181 181 340 340 340 F F F 17 340-359 180 184 184 337 346 346 F F F <--- 18 360-379 182 184 184 342 344 344 F F F <--- 19 380-399 182 184 184 342 345 345 F T T <--- 20 400-419 182 184 184 342 345 345 F T T <--- 21 420-439 180 180 180 338 338 338 F F F 22 440-459 179 184 184 335 346 346 F F F <--- 23 460-479 178 178 178 334 334 334 F F F 24 480-499 181 184 184 340 346 346 F F F <--- 25 500-519 181 181 181 340 340 340 F F F 26 520-539 177 177 177 333 333 333 F F F 27 540-559 182 182 182 341 341 341 F F F 28 560-579 182 182 182 342 342 342 F T T <--- 29 580-599 182 182 182 342 342 342 F T T <--- Max gen: 184, Max flush: 346 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 182 182 182 342 342 342 F F F 2 040-059 172 172 172 325 325 325 F F F 3 060-079 178 178 178 334 334 334 F F F 4 080-099 185 185 185 347 347 347 F F F 5 100-119 185 185 185 347 347 347 F F F 6 120-139 183 183 183 343 343 343 F F F 7 140-159 185 185 185 347 347 347 F F F 8 160-179 184 184 184 344 344 344 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 176 176 176 332 332 332 F F F 11 220-239 180 180 180 336 336 336 F F F 12 240-259 184 184 184 344 344 344 F F F 13 260-279 173 173 173 326 326 326 F F F 14 280-299 180 180 180 337 337 337 F F F 15 300-319 185 185 185 347 347 347 F F F 16 320-339 181 181 181 340 340 340 F F F 17 340-359 184 184 184 346 346 346 F F F 18 360-379 184 184 184 344 344 344 F F F 19 380-399 185 185 185 347 347 347 F F F 20 400-419 185 185 185 347 347 347 F F F 21 420-439 180 180 180 338 338 338 F F F 22 440-459 184 184 184 346 346 346 F F F 23 460-479 178 178 178 334 334 334 F F F 24 480-499 184 184 184 346 346 346 F F F 25 500-519 181 181 181 340 340 340 F F F 26 520-539 177 177 177 333 333 333 F F F 27 540-559 182 182 182 341 341 341 F F F 28 560-579 185 185 185 347 347 347 F F F 29 580-599 185 185 185 347 347 347 F F F Max gen: 185, Max flush: 347 Begin loop 93 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.431944269Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.443811062Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444134749Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444174804Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444187575Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444205124Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949} {"msg":"Crucible 2135a74e-174b-4502-b8ec-ef52b8cfe764 has session id: 51353952-b15c-49d7-b638-9b25b5f782eb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444536726Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Upstairs opts: Upstairs UUID: 2135a74e-174b-4502-b8ec-ef52b8cfe764, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444566387Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Crucible stats registered with UUID: 2135a74e-174b-4502-b8ec-ef52b8cfe764","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444583757Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:46.444606102Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444765642Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"io task","client":"2","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444790387Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"io task","client":"0","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444805597Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"io task","client":"1","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444937384Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"task":"crutest"} {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444951715Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949} {"msg":"2135a74e-174b-4502-b8ec-ef52b8cfe764 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444982106Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.444995947Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"client":"0","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.445010418Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"client":"1","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.44502426Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"client":"2","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.445038181Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"io task","client":"2","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.445052552Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"io task","client":"0","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.445066753Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"io task","client":"1","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:44601)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.445084772Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"io task","client":"0","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:36610)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.445100012Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"io task","client":"1","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:48856)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.445117121Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"io task","client":"2","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.44586731Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"client":"0","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.445895362Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"client":"1","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.445982238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"client":"2","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446242694Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446279831Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"[0]R flush_numbers[0..12]: [342, 342, 325, 334, 347, 347, 343, 347, 344, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446341232Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"[0]R generation[0..12]: [182, 182, 172, 178, 185, 185, 183, 185, 184, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446358231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446374841Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"[1]R flush_numbers[0..12]: [342, 342, 325, 334, 347, 347, 343, 347, 344, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446390661Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"[1]R generation[0..12]: [182, 182, 172, 178, 185, 185, 183, 185, 184, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446406581Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446422981Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"[2]R flush_numbers[0..12]: [342, 342, 325, 334, 347, 347, 343, 347, 344, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446438871Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"[2]R generation[0..12]: [182, 182, 172, 178, 185, 185, 183, 185, 184, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446457699Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446474239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 186","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446493177Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Generation requested: 186 >= found:186","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446507638Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Next flush: 348","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446521059Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446534611Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"":"downstairs","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446548922Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446561664Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"client":"0","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446576114Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"clientDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":"1","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446593044Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"client":"2","":"downstairs","session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446607175Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446620537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"2135a74e-174b-4502-b8ec-ef52b8cfe764 is now active with session: 51353952-b15c-49d7-b638-9b25b5f782eb","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446635327Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446648149Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949,"session_id":"51353952-b15c-49d7-b638-9b25b5f782eb"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446661741Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949} {"msg":"Activated sub_volume 2135a74e-174b-4502-b8ec-ef52b8cfe764","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:46.446673503Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 109 len 16384 data: 12 11 10 11 02/30 Read block 518 len 4096 03/30 Write block 328 len 40960 data: 15 12 12 14 13 14 16 17 18 15 04/30 Read block 362 len 12288 05/30 Write block 335 len 28672 data: 18 19 16 18 15 14 16 06/30 Read block 381 len 32768 07/30 Write block 255 len 24576 data: 17 16 14 15 14 12 08/30 Read block 168 len 8192 09/30 Read block 268 len 36864 10/30 Read block 288 len 32768 11/30 Read block 134 len 40960 12/30 Write block 328 len 12288 data: 16 13 13 13/30 Write block 575 len 20480 data: 12 13 15 16 16 14/30 Write block 246 len 4096 data: 22 15/30 Read block 12 len 28672 16/30 Read block 48 len 8192 17/30 Read block 248 len 32768 18/30 Read block 371 len 16384 19/30 Read block 16 len 8192 20/30 Write block 429 len 16384 data: 17 11 12 16 21/30 Read block 166 len 28672 22/30 Read block 62 len 32768 23/30 Write block 131 len 28672 data: 14 15 18 17 21 20 18 24/30 Read block 232 len 20480 25/30 Read block 40 len 36864 26/30 Read block 138 len 40960 27/30 Write block 353 len 32768 data: 15 17 17 18 15 14 12 14 28/30 Write block 54 len 16384 data: 15 13 9 9 29/30 Write block 311 len 28672 data: 16 18 16 14 14 11 10 30/30 Write block 52 len 4096 data: 14 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Sent false 1001 Acked Read 1 Done Done Sent false 1002 Acked Write 10 Done Done Sent false 1003 Acked Read 3 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Read 8 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Read 2 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Read 10 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Read 4 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Read 8 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Read 9 Done Done Sent false 1025 Acked Read 10 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Write 4 Sent Sent Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 30 36 Done 27 27 0 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:46.458366875Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1949} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 182 182 182 342 342 342 F F F 2 040-059 172 172 172 325 325 325 T T F <--- 3 060-079 178 178 178 334 334 334 F F F 4 080-099 185 185 185 347 347 347 F F F 5 100-119 185 185 185 347 347 347 T T F <--- 6 120-139 183 183 183 343 343 343 T T F <--- 7 140-159 185 185 185 347 347 347 F F F 8 160-179 184 184 184 344 344 344 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 176 176 176 332 332 332 F F F 11 220-239 180 180 180 336 336 336 F F F 12 240-259 184 184 184 344 344 344 T T F <--- 13 260-279 173 173 173 326 326 326 T T F <--- 14 280-299 180 180 180 337 337 337 F F F 15 300-319 185 185 185 347 347 347 T T F <--- 16 320-339 181 181 181 340 340 340 T T F <--- 17 340-359 184 184 184 346 346 346 T T F <--- 18 360-379 184 184 184 344 344 344 T T F <--- 19 380-399 185 185 185 347 347 347 F F F 20 400-419 185 185 185 347 347 347 F F F 21 420-439 180 180 180 338 338 338 T T F <--- 22 440-459 184 184 184 346 346 346 F F F 23 460-479 178 178 178 334 334 334 F F F 24 480-499 184 184 184 346 346 346 F F F 25 500-519 181 181 181 340 340 340 F F F 26 520-539 177 177 177 333 333 333 F F F 27 540-559 182 182 182 341 341 341 F F F 28 560-579 185 185 185 347 347 347 T T F <--- 29 580-599 185 185 185 347 347 347 F F F Max gen: 185, Max flush: 347 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 182 182 182 342 342 342 F F F 2 040-059 186 186 186 348 348 348 F F F 3 060-079 178 178 178 334 334 334 F F F 4 080-099 185 185 185 347 347 347 F F F 5 100-119 186 186 186 348 348 348 F F F 6 120-139 186 186 186 348 348 348 F F F 7 140-159 185 185 185 347 347 347 F F F 8 160-179 184 184 184 344 344 344 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 176 176 176 332 332 332 F F F 11 220-239 180 180 180 336 336 336 F F F 12 240-259 186 186 186 348 348 348 F F F 13 260-279 186 186 186 348 348 348 F F F 14 280-299 180 180 180 337 337 337 F F F 15 300-319 186 186 186 348 348 348 F F F 16 320-339 186 186 186 348 348 348 F F F 17 340-359 186 186 186 348 348 348 F F F 18 360-379 186 186 186 348 348 348 F F F 19 380-399 185 185 185 347 347 347 F F F 20 400-419 185 185 185 347 347 347 F F F 21 420-439 186 186 186 348 348 348 F F F 22 440-459 184 184 184 346 346 346 F F F 23 460-479 178 178 178 334 334 334 F F F 24 480-499 184 184 184 346 346 346 F F F 25 500-519 181 181 181 340 340 340 F F F 26 520-539 177 177 177 333 333 333 F F F 27 540-559 182 182 182 341 341 341 F F F 28 560-579 186 186 186 348 348 348 F F F 29 580-599 185 185 185 347 347 347 F F F Max gen: 186, Max flush: 348 Begin loop 94 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.780751756Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:50.791345109Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.791379408Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.801603313Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.801905343Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.801943019Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.801954842Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.801972291Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957} {"msg":"Crucible 3ce7d40c-0e78-4539-bb09-1c36cce895f7 has session id: 9e33be2b-173b-42cb-b2fe-d045fe58065e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802330296Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Upstairs opts: Upstairs UUID: 3ce7d40c-0e78-4539-bb09-1c36cce895f7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802364994Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Crucible stats registered with UUID: 3ce7d40c-0e78-4539-bb09-1c36cce895f7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802381134Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:50.802394526Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802552177Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"io task","client":"2","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802587504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"io task","client":"0","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802603404Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"io task","client":"1","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802703571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"task":"crutest"} {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802719252Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957} {"msg":"3ce7d40c-0e78-4539-bb09-1c36cce895f7 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.80273742Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802750582Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"client":"0","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802764693Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"client":"1","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802778614Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"client":"2","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802792566Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"io task","client":"2","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802806946Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"io task","client":"0","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802821337Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"io task","client":"1","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"ds_connection connected from Ok(127.0.0.1:39302)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802903206Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"io task","client":"0","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"ds_connection connected from Ok(127.0.0.1:40585)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.802932647Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"io task","client":"2","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"ds_connection connected from Ok(127.0.0.1:63643)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.803030696Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"io task","client":"1","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.803598809Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"client":"2","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.803634756Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"client":"1","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.803768272Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"client":"0","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.803982848Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804013958Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"[0]R flush_numbers[0..12]: [342, 342, 348, 334, 347, 348, 348, 347, 344, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804134352Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"[0]R generation[0..12]: [182, 182, 186, 178, 185, 186, 186, 185, 184, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804151601Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804168171Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"[1]R flush_numbers[0..12]: [342, 342, 348, 334, 347, 348, 348, 347, 344, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804183811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"[1]R generation[0..12]: [182, 182, 186, 178, 185, 186, 186, 185, 184, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804200001Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804216511Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"[2]R flush_numbers[0..12]: [342, 342, 348, 334, 347, 348, 348, 347, 344, 335, 332, 336]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804232171Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"[2]R generation[0..12]: [182, 182, 186, 178, 185, 186, 186, 185, 184, 179, 176, 180]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804248051Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804264431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Max found gen is 187","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2026-03-14T05:07:50.804278062Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Generation requested: 188 >= found:187","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.80429771Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Next flush: 349","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804311351Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804327131Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"":"downstairs","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804341682Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804354364Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"client":"0","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804368275Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"client":"1","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :"2026-03-14T05:07:50.804382137Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"client":"2","":"downstairs","session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804399306Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804412777Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"3ce7d40c-0e78-4539-bb09-1c36cce895f7 is now active with session: 9e33be2b-173b-42cb-b2fe-d045fe58065e","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804428008Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804441219Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957,"session_id":"9e33be2b-173b-42cb-b2fe-d045fe58065e"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804454551Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957} {"msg":"Activated sub_volume 3ce7d40c-0e78-4539-bb09-1c36cce895f7","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:50.804466244Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 438 len 36864 data: 15 15 16 16 18 18 15 14 16 02/30 Flush 03/30 Write block 521 len 20480 data: 11 13 15 13 15 04/30 Write block 213 len 4096 data: 20 05/30 Read block 332 len 36864 06/30 Write block 246 len 4096 data: 23 07/30 Flush 08/30 Flush 09/30 Write block 345 len 12288 data: 15 15 14 10/30 Read block 376 len 8192 11/30 Write block 527 len 20480 data: 17 16 14 16 18 12/30 Read block 189 len 28672 13/30 Write block 501 len 12288 data: 17 17 17 14/30 Write block 487 len 8192 data: 17 17 15/30 Write block 220 len 12288 data: 14 12 11 16/30 Write block 461 len 32768 data: 16 11 10 10 10 10 9 7 17/30 Read block 532 len 24576 18/30 Read block 531 len 20480 19/30 Flush 20/30 Write block 523 len 16384 data: 16 14 16 16 21/30 Read block 444 len 20480 22/30 Read block 90 len 36864 23/30 Write block 524 len 16384 data: 15 17 17 18 24/30 Flush 25/30 Write block 277 len 28672 data: 12 11 10 8 8 9 9 26/30 Flush 27/30 Write block 384 len 4096 data: 10 28/30 Write block 374 len 4096 data: 12 29/30 Read block 553 len 24576 30/30 Write block 463 len 16384 data: 11 11 11 11 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 5 Sent Done Done false 1003 Acked Write 1 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 3 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Write 2 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Read 5 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:50.815783422Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1957} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 182 182 182 342 342 342 F F F 2 040-059 186 186 186 348 348 348 F F F 3 060-079 178 178 178 334 334 334 F F F 4 080-099 185 185 185 347 347 347 F F F 5 100-119 186 186 186 348 348 348 F F F 6 120-139 186 186 186 348 348 348 F F F 7 140-159 185 185 185 347 347 347 F F F 8 160-179 184 184 184 344 344 344 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 176 188 188 332 350 350 F F F <--- 11 220-239 180 188 188 336 352 352 F F F <--- 12 240-259 186 188 188 348 350 350 F F F <--- 13 260-279 186 188 188 348 354 354 F F F <--- 14 280-299 180 188 188 337 354 354 F F F <--- 15 300-319 186 186 186 348 348 348 F F F 16 320-339 186 186 186 348 348 348 F F F 17 340-359 186 188 188 348 352 352 F F F <--- 18 360-379 186 186 186 348 348 348 F T T <--- 19 380-399 185 185 185 347 347 347 F T T <--- 20 400-419 185 185 185 347 347 347 F F F 21 420-439 186 188 188 348 349 349 T F F <--- 22 440-459 184 188 188 346 349 349 T F F <--- 23 460-479 178 188 188 334 352 352 F T T <--- 24 480-499 184 188 188 346 352 352 F F F <--- 25 500-519 181 188 188 340 352 352 F F F <--- 26 520-539 177 188 188 333 353 353 F F F <--- 27 540-559 182 182 182 341 341 341 F F F 28 560-579 186 186 186 348 348 348 F F F 29 580-599 185 185 185 347 347 347 F F F Max gen: 188, Max flush: 354 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 182 182 182 342 342 342 F F F 2 040-059 186 186 186 348 348 348 F F F 3 060-079 178 178 178 334 334 334 F F F 4 080-099 185 185 185 347 347 347 F F F 5 100-119 186 186 186 348 348 348 F F F 6 120-139 186 186 186 348 348 348 F F F 7 140-159 185 185 185 347 347 347 F F F 8 160-179 184 184 184 344 344 344 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 188 188 188 350 350 350 F F F 11 220-239 188 188 188 352 352 352 F F F 12 240-259 188 188 188 350 350 350 F F F 13 260-279 188 188 188 354 354 354 F F F 14 280-299 188 188 188 354 354 354 F F F 15 300-319 186 186 186 348 348 348 F F F 16 320-339 186 186 186 348 348 348 F F F 17 340-359 188 188 188 352 352 352 F F F 18 360-379 189 189 189 355 355 355 F F F 19 380-399 189 189 189 355 355 355 F F F 20 400-419 185 185 185 347 347 347 F F F 21 420-439 188 188 188 349 349 349 F F F 22 440-459 188 188 188 349 349 349 F F F 23 460-479 189 189 189 355 355 355 F F F 24 480-499 188 188 188 352 352 352 F F F 25 500-519 188 188 188 352 352 352 F F F 26 520-539 188 188 188 353 353 353 F F F 27 540-559 182 182 182 341 341 341 F F F 28 560-579 186 186 186 348 348 348 F F F 29 580-599 185 185 185 347 347 347 F F F Max gen: 189, Max flush: 355 Begin loop 95 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.149259025Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.161052803Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.161372033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.161410099Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.161422491Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.16144032Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965} {"msg":"Crucible e3d38842-b53a-4afe-9aea-1fa913663f3f has session id: e9d6f6f0-71cb-4939-a86c-901e4db7dd5a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.161798665Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Upstairs opts: Upstairs UUID: e3d38842-b53a-4afe-9aea-1fa913663f3f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.161833443Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Crucible stats registered with UUID: e3d38842-b53a-4afe-9aea-1fa913663f3f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.161849373Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:55.161862705Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162019466Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"io task","client":"2","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162053655Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"io task","client":"0","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162069854Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"io task","client":"1","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162178156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"task":"crutest"} {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162192707Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965} {"msg":"e3d38842-b53a-4afe-9aea-1fa913663f3f active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162212275Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162225137Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"client":"0","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162239118Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"client":"1","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162253069Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"client":"2","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.16226755Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"io task","client":"2","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162281861Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"io task","client":"0","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.16230033Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"io task","client":"1","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"ds_connection connected from Ok(127.0.0.1:64536)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162340125Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"io task","client":"1","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"ds_connection connected from Ok(127.0.0.1:56501)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162492719Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"io task","client":"2","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"ds_connection connected from Ok(127.0.0.1:45666)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.162520211Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"io task","client":"0","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163098028Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"client":"2","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163127839Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"client":"0","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163145029Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"client":"1","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163456883Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163471663Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"[0]R flush_numbers[0..12]: [342, 342, 348, 334, 347, 348, 348, 347, 344, 335, 350, 352]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163496778Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"[0]R generation[0..12]: [182, 182, 186, 178, 185, 186, 186, 185, 184, 179, 188, 188]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163513567Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163530397Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"[1]R flush_numbers[0..12]: [342, 342, 348, 334, 347, 348, 348, 347, 344, 335, 350, 352]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163546297Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"[1]R generation[0..12]: [182, 182, 186, 178, 185, 186, 186, 185, 184, 179, 188, 188]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163562047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163581434Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"[2]R flush_numbers[0..12]: [342, 342, 348, 334, 347, 348, 348, 347, 344, 335, 350, 352]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163597354Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"[2]R generation[0..12]: [182, 182, 186, 178, 185, 186, 186, 185, 184, 179, 188, 188]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163612995Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163629414Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163643286Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163663133Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Next flush: 356","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163676675Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163690466Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"":"downstairs","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163704517Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163717459Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"client":"0","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.16373174Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"client":"1","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163746101Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"client":"2","":"downstairs","session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.16376281Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163776182Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"e3d38842-b53a-4afe-9aea-1fa913663f3f is now active with session: e9d6f6f0-71cb-4939-a86c-901e4db7dd5a","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163791492Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163805933Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965,"session_id":"e9d6f6f0-71cb-4939-a86c-901e4db7dd5a"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163819495Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965} {"msg":"Activated sub_volume e3d38842-b53a-4afe-9aea-1fa913663f3f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:55.163830928Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 520 len 12288 data: 13 12 14 02/30 Read block 245 len 16384 03/30 Flush 04/30 Flush 05/30 Flush 06/30 Write block 456 len 32768 data: 19 17 19 19 17 17 12 12 07/30 Write block 162 len 28672 data: 12 13 11 12 12 13 12 08/30 Write block 417 len 8192 data: 9 15 09/30 Read block 445 len 4096 10/30 Write block 43 len 12288 data: 9 8 10 11/30 Read block 411 len 8192 12/30 Read block 454 len 24576 13/30 Write block 542 len 12288 data: 9 6 6 14/30 Read block 356 len 24576 15/30 Write block 573 len 32768 data: 10 9 13 14 16 17 17 18 16/30 Write block 235 len 20480 data: 24 23 20 25 22 17/30 Read block 300 len 32768 18/30 Write block 430 len 36864 data: 12 13 17 16 17 17 18 17 16 19/30 Write block 492 len 24576 data: 16 18 18 20 16 14 20/30 Read block 63 len 20480 21/30 Write block 424 len 4096 data: 15 22/30 Read block 51 len 16384 23/30 Flush 24/30 Write block 41 len 40960 data: 11 10 10 9 11 5 8 9 10 11 25/30 Read block 136 len 20480 26/30 Write block 79 len 36864 data: 15 14 13 13 13 13 16 18 16 27/30 Read block 488 len 40960 28/30 Write block 562 len 8192 data: 7 7 29/30 Write block 339 len 8192 data: 16 15 30/30 Write block 415 len 36864 data: 10 11 10 16 14 14 17 16 17 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Read 4 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 8 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Read 1 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Read 2 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Write 1 Done Sent Done false 1021 Acked Read 4 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Write 2 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:55.176177599Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1965} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 182 182 182 342 342 342 F F F 2 040-059 190 186 190 359 348 359 T F T <--- 3 060-079 178 178 178 334 334 334 T F T <--- 4 080-099 185 185 185 347 347 347 T F T <--- 5 100-119 186 186 186 348 348 348 F F F 6 120-139 186 186 186 348 348 348 F F F 7 140-159 185 185 185 347 347 347 F F F 8 160-179 190 184 190 359 344 359 F F F <--- 9 180-199 179 179 179 335 335 335 F F F 10 200-219 188 188 188 350 350 350 F F F 11 220-239 190 188 190 359 352 359 F F F <--- 12 240-259 188 188 188 350 350 350 F F F 13 260-279 188 188 188 354 354 354 F F F 14 280-299 188 188 188 354 354 354 F F F 15 300-319 186 186 186 348 348 348 F F F 16 320-339 186 186 186 348 348 348 T F T <--- 17 340-359 188 188 188 352 352 352 T F T <--- 18 360-379 189 189 189 355 355 355 F F F 19 380-399 189 189 189 355 355 355 F F F 20 400-419 190 185 190 359 347 359 T F T <--- 21 420-439 190 188 190 359 349 359 T F T <--- 22 440-459 190 188 190 359 349 359 F F F <--- 23 460-479 190 189 190 359 355 359 F F F <--- 24 480-499 190 188 190 359 352 359 F F F <--- 25 500-519 188 188 188 352 352 352 F F F 26 520-539 190 188 190 356 353 356 F F F <--- 27 540-559 190 182 190 359 341 359 F F F <--- 28 560-579 190 186 190 359 348 359 T F T <--- 29 580-599 190 185 190 359 347 359 F F F <--- Max gen: 190, Max flush: 359 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 95 Verifying data now No change after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 182 182 182 342 342 342 F F F 2 040-059 191 191 191 360 360 360 F F F 3 060-079 191 191 191 360 360 360 F F F 4 080-099 191 191 191 360 360 360 F F F 5 100-119 186 186 186 348 348 348 F F F 6 120-139 186 186 186 348 348 348 F F F 7 140-159 185 185 185 347 347 347 F F F 8 160-179 190 190 190 359 359 359 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 188 188 188 350 350 350 F F F 11 220-239 190 190 190 359 359 359 F F F 12 240-259 188 188 188 350 350 350 F F F 13 260-279 188 188 188 354 354 354 F F F 14 280-299 188 188 188 354 354 354 F F F 15 300-319 186 186 186 348 348 348 F F F 16 320-339 191 191 191 360 360 360 F F F 17 340-359 191 191 191 360 360 360 F F F 18 360-379 189 189 189 355 355 355 F F F 19 380-399 189 189 189 355 355 355 F F F 20 400-419 191 191 191 360 360 360 F F F 21 420-439 191 191 191 360 360 360 F F F 22 440-459 190 190 190 359 359 359 F F F 23 460-479 190 190 190 359 359 359 F F F 24 480-499 190 190 190 359 359 359 F F F 25 500-519 188 188 188 352 352 352 F F F 26 520-539 190 190 190 356 356 356 F F F 27 540-559 190 190 190 359 359 359 F F F 28 560-579 191 191 191 360 360 360 F F F 29 580-599 190 190 190 359 359 359 F F F Max gen: 191, Max flush: 360 Begin loop 96 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.511319165Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.522923543Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.523236047Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.523275572Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.523288434Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.523306282Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973} {"msg":"Crucible 3903231e-5b6b-467c-b16a-c5c235082680 has session id: 8942d7de-f84d-44e8-8f08-71801d999ce8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.523649917Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Upstairs opts: Upstairs UUID: 3903231e-5b6b-467c-b16a-c5c235082680, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.523685064Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Crucible stats registered with UUID: 3903231e-5b6b-467c-b16a-c5c235082680","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.523701244Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:59.523715106Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.523850171Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"io task","client":"2","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.523880811Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"io task","client":"0","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.523895912Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"io task","client":"1","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524017306Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"task":"crutest"} {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524031197Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973} {"msg":"3903231e-5b6b-467c-b16a-c5c235082680 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524052623Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524065665Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"client":"0","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524079447Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"client":"1","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524093318Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"client":"2","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524106869Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"io task","client":"2","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.52412113Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"io task","client":"0","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524140878Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"io task","client":"1","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"ds_connection connected from Ok(127.0.0.1:52987)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524179584Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"io task","client":"1","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"ds_connection connected from Ok(127.0.0.1:50641)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524195634Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"io task","client":"0","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"ds_connection connected from Ok(127.0.0.1:47161)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524211074Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"io task","client":"2","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524920228Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"client":"1","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524948131Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"client":"2","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.524968178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"client":"0","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525292494Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525338156Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"[0]R flush_numbers[0..12]: [342, 342, 360, 360, 360, 348, 348, 347, 359, 335, 350, 359]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525407132Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"[0]R generation[0..12]: [182, 182, 191, 191, 191, 186, 186, 185, 190, 179, 188, 190]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525424062Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525440501Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"[1]R flush_numbers[0..12]: [342, 342, 360, 360, 360, 348, 348, 347, 359, 335, 350, 359]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525456411Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"[1]R generation[0..12]: [182, 182, 191, 191, 191, 186, 186, 185, 190, 179, 188, 190]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525472471Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525488491Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"[2]R flush_numbers[0..12]: [342, 342, 360, 360, 360, 348, 348, 347, 359, 335, 350, 359]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525504341Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"[2]R generation[0..12]: [182, 182, 191, 191, 191, 186, 186, 185, 190, 179, 188, 190]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525520191Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525536431Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525549703Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs",Wait for a query_work_queue command to finish before sending IO "":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525570759Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Next flush: 361","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.52558655Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525600121Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"":"downstairs","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525614352Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525626754Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"client":"0","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525640985Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"client":"1","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525655006Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"client":"2","":"downstairs","session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2026-03-14T05:07:59.525669617Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525685327Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"3903231e-5b6b-467c-b16a-c5c235082680 is now active with session: 8942d7de-f84d-44e8-8f08-71801d999ce8","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525700178Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525713959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973,"session_id":"8942d7de-f84d-44e8-8f08-71801d999ce8"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525727341Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973} {"msg":"Activated sub_volume 3903231e-5b6b-467c-b16a-c5c235082680","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:07:59.525738604Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 418 len 8192 data: 17 15 02/30 Write block 72 len 12288 data: 15 14 18 03/30 Read block 317 len 4096 04/30 Read block 230 len 12288 05/30 Flush 06/30 Write block 477 len 24576 data: 16 17 18 17 15 16 07/30 Read block 10 len 8192 08/30 Read block 322 len 12288 09/30 Read block 16 len 32768 10/30 Write block 27 len 12288 data: 11 15 14 11/30 Read block 189 len 12288 12/30 Write block 546 len 8192 data: 10 16 13/30 Write block 454 len 8192 data: 16 18 14/30 Read block 417 len 12288 15/30 Write block 367 len 40960 data: 21 22 22 21 20 16 13 13 14 14 16/30 Write block 232 len 36864 data: 18 18 20 25 24 21 26 23 21 17/30 Read block 402 len 28672 18/30 Read block 147 len 32768 19/30 Read block 99 len 16384 20/30 Read block 163 len 24576 21/30 Read block 80 len 24576 22/30 Read block 18 len 16384 23/30 Write block 201 len 12288 data: 13 12 17 24/30 Write block 230 len 20480 data: 13 15 19 19 21 25/30 Write block 513 len 40960 data: 13 14 17 13 14 13 15 14 13 15 26/30 Write block 91 len 12288 data: 14 12 11 27/30 Read block 51 len 24576 28/30 Read block 107 len 24576 29/30 Write block 587 len 28672 data: 13 11 13 11 10 13 13 30/30 Write block 66 len 8192 data: 14 14 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Sent false 1001 Acked Write 3 Done Done Sent false 1002 Acked Read 1 Done Done Sent false 1003 Acked Read 3 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Read 2 Done Done Sent false 1007 Acked Read 3 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Read 3 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Read 3 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 30 34 Done 28 28 0 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) Some(JobId(1004)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:07:59.536657782Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1973} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 182 182 182 342 342 342 T T F <--- 2 040-059 191 191 191 360 360 360 F F F 3 060-079 192 192 191 361 361 360 T T F <--- 4 080-099 191 191 191 360 360 360 T T F <--- 5 100-119 186 186 186 348 348 348 F F F 6 120-139 186 186 186 348 348 348 F F F 7 140-159 185 185 185 347 347 347 F F F 8 160-179 190 190 190 359 359 359 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 188 188 188 350 350 350 T T F <--- 11 220-239 190 190 190 359 359 359 T T F <--- 12 240-259 188 188 188 350 350 350 T T F <--- 13 260-279 188 188 188 354 354 354 F F F 14 280-299 188 188 188 354 354 354 F F F 15 300-319 186 186 186 348 348 348 F F F 16 320-339 191 191 191 360 360 360 F F F 17 340-359 191 191 191 360 360 360 F F F 18 360-379 189 189 189 355 355 355 T T F <--- 19 380-399 189 189 189 355 355 355 F F F 20 400-419 192 192 191 361 361 360 F F F <--- 21 420-439 191 191 191 360 360 360 F F F 22 440-459 190 190 190 359 359 359 T T F <--- 23 460-479 190 190 190 359 359 359 T T F <--- 24 480-499 190 190 190 359 359 359 T T F <--- 25 500-519 188 188 188 352 352 352 T T F <--- 26 520-539 190 190 190 356 356 356 T T F <--- 27 540-559 190 190 190 359 359 359 T T F <--- 28 560-579 191 191 191 360 360 360 F F F 29 580-599 190 190 190 359 359 359 T T F <--- Max gen: 192, Max flush: 361 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 193 193 193 362 362 362 F F F 2 040-059 191 191 191 360 360 360 F F F 3 060-079 193 193 193 362 362 362 F F F 4 080-099 193 193 193 362 362 362 F F F 5 100-119 186 186 186 348 348 348 F F F 6 120-139 186 186 186 348 348 348 F F F 7 140-159 185 185 185 347 347 347 F F F 8 160-179 190 190 190 359 359 359 F F F 9 180-199 179 179 179 335 335 335 F F F 10 200-219 193 193 193 362 362 362 F F F 11 220-239 193 193 193 362 362 362 F F F 12 240-259 193 193 193 362 362 362 F F F 13 260-279 188 188 188 354 354 354 F F F 14 280-299 188 188 188 354 354 354 F F F 15 300-319 186 186 186 348 348 348 F F F 16 320-339 191 191 191 360 360 360 F F F 17 340-359 191 191 191 360 360 360 F F F 18 360-379 193 193 193 362 362 362 F F F 19 380-399 189 189 189 355 355 355 F F F 20 400-419 192 192 192 361 361 361 F F F 21 420-439 191 191 191 360 360 360 F F F 22 440-459 193 193 193 362 362 362 F F F 23 460-479 193 193 193 362 362 362 F F F 24 480-499 193 193 193 362 362 362 F F F 25 500-519 193 193 193 362 362 362 F F F 26 520-539 193 193 193 362 362 362 F F F 27 540-559 193 193 193 362 362 362 F F F 28 560-579 191 191 191 360 360 360 F F F 29 580-599 193 193 193 362 362 362 F F F Max gen: 193, Max flush: 362 Begin loop 97 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.870198825Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:08:03.880785751Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.880833382Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.89125837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.891550227Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.891589932Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.891601854Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.891619473Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981} {"msg":"Crucible a662900d-4d96-4434-aa4f-06b5d267c2fe has session id: 5c196c49-772d-40bf-b48b-ed0b3d9089e5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.891986533Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Upstairs opts: Upstairs UUID: a662900d-4d96-4434-aa4f-06b5d267c2fe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892021571Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Crucible stats registered with UUID: a662900d-4d96-4434-aa4f-06b5d267c2fe","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892037411Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:08:03.892050662Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892201048Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"io task","client":"0","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892225602Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"io task","client":"2","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892240463Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"io task","client":"1","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892358389Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"task":"crutest"} {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.89237263Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981} {"msg":"a662900d-4d96-4434-aa4f-06b5d267c2fe active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892390559Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892403351Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"client":"0","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.89242013Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"client":"1","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892433752Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"client":"2","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892447713Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"io task","client":"2","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892462174Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"io task","client":"0","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892476035Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"io task","client":"1","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:39016)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892545971Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"io task","client":"1","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:64781)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892573504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"io task","client":"2","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"ds_connection connected from Ok(127.0.0.1:47760)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.892589694Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"io task","client":"0","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893220287Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"client":"2","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893248959Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"client":"1","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893381986Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"client":"0","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893618297Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.89366159Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"[0]R flush_numbers[0..12]: [342, 362, 360, 362, 362, 348, 348, 347, 359, 335, 362, 362]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893759778Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"[0]R generation[0..12]: [182, 193, 191, 193, 193, 186, 186, 185, 190, 179, 193, 193]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893776978Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893796495Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"[1]R flush_numbers[0..12]: [342, 362, 360, 362, 362, 348, 348, 347, 359, 335, 362, 362]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893812305Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"[1]R generation[0..12]: [182, 193, 191, 193, 193, 186, 186, 185, 190, 179, 193, 193]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893827686Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893843945Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"[2]R flush_numbers[0..12]: [342, 362, 360, 362, 362, 348, 348, 347, 359, 335, 362, 362]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893859586Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"[2]R generation[0..12]: [182, 193, 191, 193, 193, 186, 186, 185, 190, 179, 193, 193]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893875246Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893891645Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Wait for a query_work_queue command to finish before sending IO Max found gen is 194","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893910863Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893925404Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Next flush: 363","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893938826Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893952137Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"":"downstairs","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893966199Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893978921Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"client":"0","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.893992902Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 1981,"client":"1","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.894012859Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"client":"2","":"downstairs","session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.89402732Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.894040652Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"a662900d-4d96-4434-aa4f-06b5d267c2fe is now active with session: 5c196c49-772d-40bf-b48b-ed0b3d9089e5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.894055732Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.894068165Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981,"session_id":"5c196c49-772d-40bf-b48b-ed0b3d9089e5"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.894081246Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981} {"msg":"Activated sub_volume a662900d-4d96-4434-aa4f-06b5d267c2fe","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:03.894092839Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 120 len 8192 data: 11 8 02/30 Read block 60 len 24576 03/30 Read block 73 len 32768 04/30 Write block 234 len 16384 data: 22 26 25 22 05/30 Read block 427 len 36864 06/30 Read block 495 len 32768 07/30 Read block 542 len 40960 08/30 Write block 23 len 24576 data: 11 10 10 7 12 16 09/30 Write block 322 len 8192 data: 11 11 10/30 Read block 67 len 36864 11/30 Write block 24 len 36864 data: 11 11 8 13 17 15 14 13 11 12/30 Write block 424 len 24576 data: 16 14 11 12 15 18 13/30 Write block 181 len 28672 data: 9 10 12 13 13 14 15 14/30 Read block 351 len 24576 15/30 Read block 417 len 32768 16/30 Write block 530 len 4096 data: 17 17/30 Read block 181 len 40960 18/30 Write block 423 len 24576 data: 18 17 15 12 13 16 19/30 Write block 531 len 8192 data: 19 17 20/30 Read block 172 len 28672 21/30 Write block 66 len 36864 data: 15 15 16 19 20 18 16 15 19 22/30 Write block 96 len 32768 data: 13 12 7 10 13 17 19 19 23/30 Write block 545 len 40960 data: 9 11 17 17 16 18 15 14 13 14 24/30 Write block 310 len 12288 data: 16 17 19 25/30 Read block 186 len 40960 26/30 Read block 328 len 36864 27/30 Read block 515 len 4096 28/30 Write block 174 len 8192 data: 13 14 29/30 Read block 144 len 4096 30/30 Write block 220 len 24576 data: 15 13 12 14 16 10 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Sent Done Done false 1001 Acked Read 6 Sent Done Done false 1002 Acked Read 8 Sent Done Done false 1003 Acked Write 4 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 10 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Read 10 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:08:03.905710138Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1981} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 193 193 193 362 362 362 F T T <--- 2 040-059 191 191 191 360 360 360 F F F 3 060-079 193 193 193 362 362 362 F T T <--- 4 080-099 193 193 193 362 362 362 F T T <--- 5 100-119 186 186 186 348 348 348 F T T <--- 6 120-139 186 186 186 348 348 348 F T T <--- 7 140-159 185 185 185 347 347 347 F F F 8 160-179 190 190 190 359 359 359 F T T <--- 9 180-199 179 179 179 335 335 335 F T T <--- 10 200-219 193 193 193 362 362 362 F F F 11 220-239 193 193 193 362 362 362 F T T <--- 12 240-259 193 193 193 362 362 362 F F F 13 260-279 188 188 188 354 354 354 F F F 14 280-299 188 188 188 354 354 354 F F F 15 300-319 186 186 186 348 348 348 F T T <--- 16 320-339 191 191 191 360 360 360 F T T <--- 17 340-359 191 191 191 360 360 360 F F F 18 360-379 193 193 193 362 362 362 F F F 19 380-399 189 189 189 355 355 355 F F F 20 400-419 192 192 192 361 361 361 F F F 21 420-439 191 191 191 360 360 360 F T T <--- 22 440-459 193 193 193 362 362 362 F F F 23 460-479 193 193 193 362 362 362 F F F 24 480-499 193 193 193 362 362 362 F F F 25 500-519 193 193 193 362 362 362 F F F 26 520-539 193 193 193 362 362 362 F T T <--- 27 540-559 193 193 193 362 362 362 F T T <--- 28 560-579 191 191 191 360 360 360 F F F 29 580-599 193 193 193 362 362 362 F F F Max gen: 193, Max flush: 362 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 194 194 194 363 363 363 F F F 2 040-059 191 191 191 360 360 360 F F F 3 060-079 194 194 194 363 363 363 F F F 4 080-099 194 194 194 363 363 363 F F F 5 100-119 194 194 194 363 363 363 F F F 6 120-139 194 194 194 363 363 363 F F F 7 140-159 185 185 185 347 347 347 F F F 8 160-179 194 194 194 363 363 363 F F F 9 180-199 194 194 194 363 363 363 F F F 10 200-219 193 193 193 362 362 362 F F F 11 220-239 194 194 194 363 363 363 F F F 12 240-259 193 193 193 362 362 362 F F F 13 260-279 188 188 188 354 354 354 F F F 14 280-299 188 188 188 354 354 354 F F F 15 300-319 194 194 194 363 363 363 F F F 16 320-339 194 194 194 363 363 363 F F F 17 340-359 191 191 191 360 360 360 F F F 18 360-379 193 193 193 362 362 362 F F F 19 380-399 189 189 189 355 355 355 F F F 20 400-419 192 192 192 361 361 361 F F F 21 420-439 194 194 194 363 363 363 F F F 22 440-459 193 193 193 362 362 362 F F F 23 460-479 193 193 193 362 362 362 F F F 24 480-499 193 193 193 362 362 362 F F F 25 500-519 193 193 193 362 362 362 F F F 26 520-539 194 194 194 363 363 363 F F F 27 540-559 194 194 194 363 363 363 F F F 28 560-579 191 191 191 360 360 360 F F F 29 580-599 193 193 193 362 362 362 F F F Max gen: 194, Max flush: 363 Begin loop 98 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.23315627Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.244734504Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245056442Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245097586Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245110538Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245128476Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989} {"msg":"Crucible 66a54723-937b-41c7-91ad-773f3d28e9e5 has session id: 36be462f-0b5d-45ce-b87f-5b84e9a15192","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245442139Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Upstairs opts: Upstairs UUID: 66a54723-937b-41c7-91ad-773f3d28e9e5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245477337Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Crucible stats registered with UUID: 66a54723-937b-41c7-91ad-773f3d28e9e5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245495616Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:08:08.245508827Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245665009Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"io task","client":"2","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245689254Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"io task","client":"0","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245704245Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"io task","client":"1","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245825648Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"task":"crutest"} {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245841238Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989} {"msg":"66a54723-937b-41c7-91ad-773f3d28e9e5 active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245862225Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245875007Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"client":"0","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245889288Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"client":"1","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.24590318Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"client":"2","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245917541Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"io task","client":"2","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245931772Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"io task","client":"0","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245946063Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"io task","client":"1","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"ds_connection connected from Ok(127.0.0.1:59211)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245961133Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"io task","client":"1","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"ds_connection connected from Ok(127.0.0.1:35538)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245976144Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"io task","client":"0","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"ds_connection connected from Ok(127.0.0.1:48172)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.245993043Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"io task","client":"2","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.24674544Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"client":"0","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.246774362Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"client":"1","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.246797647Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"client":"2","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247148757Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247163537Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"[0]R flush_numbers[0..12]: [342, 363, 360, 363, 363, 363, 363, 347, 363, 363, 362, 363]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247189231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"[0]R generation[0..12]: [182, 194, 191, 194, 194, 194, 194, 185, 194, 194, 193, 194]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247205231Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247222061Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"[1]R flush_numbers[0..12]: [342, 363, 360, 363, 363, 363, 363, 347, 363, 363, 362, 363]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247238011Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"[1]R generation[0..12]: [182, 194, 191, 194, 194, 194, 194, 185, 194, 194, 193, 194]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247253911Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.24727031Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"[2]R flush_numbers[0..12]: [342, 363, 360, 363, 363, 363, 363, 347, 363, 363, 362, 363]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.24728646Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"[2]R generation[0..12]: [182, 194, 191, 194, 194, 194, 194, 185, 194, 194, 193, 194]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.24730239Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247321448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Max found gen is 195","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.2473353Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Generation requested: 196 >= found:195","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.24734975Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Next flush: 364","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2026-03-14T05:08:08.247363452Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247380511Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"":"downstairs","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247396841Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247409343Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"client":"0","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247423424Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"client":"1","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247437265Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"client":"2","":"downstairs","session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247451447Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247465198Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"66a54723-937b-41c7-91ad-773f3d28e9e5 is now active with session: 36be462f-0b5d-45ce-b87f-5b84e9a15192","v":0,"name":"crucible","level":30,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 time":"2026-03-14T05:08:08.247480478Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247496089Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989,"session_id":"36be462f-0b5d-45ce-b87f-5b84e9a15192"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.24750942Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989} {"msg":"Activated sub_volume 66a54723-937b-41c7-91ad-773f3d28e9e5","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:08.247520983Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 427 len 12288 data: 14 17 19 03/30 Read block 209 len 40960 04/30 Write block 298 len 36864 data: 12 11 11 10 10 9 9 9 8 05/30 Read block 108 len 12288 06/30 Read block 336 len 8192 07/30 Read block 445 len 20480 08/30 Read block 278 len 28672 09/30 Write block 155 len 12288 data: 16 16 18 10/30 Write block 39 len 16384 data: 12 13 12 11 11/30 Write block 391 len 12288 data: 12 12 13 12/30 Write block 111 len 4096 data: 11 13/30 Write block 251 len 8192 data: 25 21 14/30 Read block 524 len 40960 15/30 Read block 52 len 36864 16/30 Write block 422 len 20480 data: 17 19 18 16 13 17/30 Write block 303 len 20480 data: 10 10 10 9 12 18/30 Write block 301 len 40960 data: 11 11 11 11 11 10 13 16 15 17 19/30 Read block 422 len 36864 20/30 Write block 65 len 28672 data: 11 16 16 17 20 21 19 21/30 Read block 304 len 16384 22/30 Read block 138 len 40960 23/30 Read block 257 len 28672 24/30 Write block 525 len 20480 data: 18 18 19 17 15 25/30 Flush 26/30 Write block 302 len 28672 data: 12 12 12 12 11 14 17 27/30 Read block 484 len 32768 28/30 Write block 566 len 4096 data: 10 29/30 Write block 322 len 16384 data: 12 12 11 11 30/30 Write block 521 len 36864 data: 14 16 17 16 19 19 20 18 16 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 9 Done Done Done false 1004 Acked Read 3 Done Done Sent false 1005 Acked Read 2 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Write 4 Done Done Sent false 1010 Acked Write 3 Done Done Sent false 1011 Acked Write 1 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Write 5 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Read 4 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Write 1 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 26 28 Done 28 28 3 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:08:08.259483465Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1989} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 196 196 194 365 365 363 F F F <--- 2 040-059 196 196 191 365 365 360 F F F <--- 3 060-079 196 196 194 365 365 363 F F F <--- 4 080-099 194 194 194 363 363 363 F F F 5 100-119 196 196 194 365 365 363 F F F <--- 6 120-139 194 194 194 363 363 363 F F F 7 140-159 196 196 185 365 365 347 F F F <--- 8 160-179 194 194 194 363 363 363 F F F 9 180-199 194 194 194 363 363 363 F F F 10 200-219 193 193 193 362 362 362 F F F 11 220-239 194 194 194 363 363 363 F F F 12 240-259 196 196 193 365 365 362 F F F <--- 13 260-279 188 188 188 354 354 354 F F F 14 280-299 196 196 188 365 365 354 F F T <--- 15 300-319 196 196 194 365 365 363 T T T <--- 16 320-339 194 194 194 363 363 363 T T F <--- 17 340-359 191 191 191 360 360 360 F F F 18 360-379 193 193 193 362 362 362 F F F 19 380-399 196 196 189 365 365 355 F F F <--- 20 400-419 192 192 192 361 361 361 F F F 21 420-439 196 196 194 365 365 363 F F T <--- 22 440-459 193 193 193 362 362 362 F F F 23 460-479 193 193 193 362 362 362 F F F 24 480-499 193 193 193 362 362 362 F F F 25 500-519 193 193 193 362 362 362 F F F 26 520-539 196 196 194 365 365 363 T T F <--- 27 540-559 194 194 194 363 363 363 F F F 28 560-579 191 191 191 360 360 360 T T F <--- 29 580-599 193 193 193 362 362 362 F F F Max gen: 196, Max flush: 365 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 196 196 196 365 365 365 F F F 2 040-059 196 196 196 365 365 365 F F F 3 060-079 196 196 196 365 365 365 F F F 4 080-099 194 194 194 363 363 363 F F F 5 100-119 196 196 196 365 365 365 F F F 6 120-139 194 194 194 363 363 363 F F F 7 140-159 196 196 196 365 365 365 F F F 8 160-179 194 194 194 363 363 363 F F F 9 180-199 194 194 194 363 363 363 F F F 10 200-219 193 193 193 362 362 362 F F F 11 220-239 194 194 194 363 363 363 F F F 12 240-259 196 196 196 365 365 365 F F F 13 260-279 188 188 188 354 354 354 F F F 14 280-299 196 196 196 365 365 365 F F F 15 300-319 197 197 197 366 366 366 F F F 16 320-339 197 197 197 366 366 366 F F F 17 340-359 191 191 191 360 360 360 F F F 18 360-379 193 193 193 362 362 362 F F F 19 380-399 196 196 196 365 365 365 F F F 20 400-419 192 192 192 361 361 361 F F F 21 420-439 196 196 196 365 365 365 F F F 22 440-459 193 193 193 362 362 362 F F F 23 460-479 193 193 193 362 362 362 F F F 24 480-499 193 193 193 362 362 362 F F F 25 500-519 193 193 193 362 362 362 F F F 26 520-539 197 197 197 366 366 366 F F F 27 540-559 194 194 194 363 363 363 F F F 28 560-579 197 197 197 366 366 366 F F F 29 580-599 193 193 193 362 362 362 F F F Max gen: 197, Max flush: 366 Begin loop 99 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.584457721Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.596183272Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.5964887Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"092b5d8f3c87943d2bae7b326a9698bf1dd177a4\",\n git_commit_timestamp: \"2026-03-14T04:42:42.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.596525647Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.596537709Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.596555538Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997} {"msg":"Crucible 776f6655-2f6f-482b-b031-dc6c3abadcaa has session id: 510d0a02-963e-44cd-b5cc-cdd00548b94f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.596884312Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Upstairs opts: Upstairs UUID: 776f6655-2f6f-482b-b031-dc6c3abadcaa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.596911195Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Crucible stats registered with UUID: 776f6655-2f6f-482b-b031-dc6c3abadcaa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.596925806Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:08:12.596938947Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597095209Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"io task","client":"1","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597119164Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"io task","client":"0","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597133945Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"io task","client":"2","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597258277Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"task":"crutest"} {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597274916Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997} {"msg":"776f6655-2f6f-482b-b031-dc6c3abadcaa active request set","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597292555Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597305267Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"client":"0","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597319018Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"client":"1","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.59733269Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"client":"2","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597346381Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"io task","client":"2","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597360213Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"io task","client":"0","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597374304Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"io task","client":"1","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"ds_connection connected from Ok(127.0.0.1:63294)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597446169Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"io task","client":"0","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"ds_connection connected from Ok(127.0.0.1:59907)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597463038Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"io task","client":"2","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"ds_connection connected from Ok(127.0.0.1:54744)","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.597578725Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"io task","client":"1","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598171183Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"client":"2","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598190111Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"client":"0","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598206201Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"client":"1","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598532496Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598579916Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"[0]R flush_numbers[0..12]: [342, 365, 365, 365, 363, 365, 363, 365, 363, 363, 362, 363]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598624598Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"[0]R generation[0..12]: [182, 196, 196, 196, 194, 196, 194, 196, 194, 194, 193, 194]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598642087Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598658747Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"[1]R flush_numbers[0..12]: [342, 365, 365, 365, 363, 365, 363, 365, 363, 363, 362, 363]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598674687Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"[1]R generation[0..12]: [182, 196, 196, 196, 194, 196, 194, 196, 194, 194, 193, 194]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598690547Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598706616Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"[2]R flush_numbers[0..12]: [342, 365, 365, 365, 363, 365, 363, 365, 363, 363, 362, 363]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598722457Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"[2]R generation[0..12]: [182, 196, 196, 196, 194, 196, 194, 196, 194, 194, 193, 194]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598737837Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598754067Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598767748Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHHWait for a query_work_queue command to finish before sending IO ","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Generation requested: 198 >= found:198","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598787326Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Next flush: 367","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598800637Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598813949Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"":"downstairs","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.59882762Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598842151Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"client":"0","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598856522Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"client":"1","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598870763Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 1997,"client":"2","":"downstairs","session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598889132Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598902853Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"776f6655-2f6f-482b-b031-dc6c3abadcaa is now active with session: 510d0a02-963e-44cd-b5cc-cdd00548b94f","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598917684Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598930426Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997,"session_id":"510d0a02-963e-44cd-b5cc-cdd00548b94f"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598943448Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997} {"msg":"Activated sub_volume 776f6655-2f6f-482b-b031-dc6c3abadcaa","v":0,"name":"crucible","level":30,"time":"2026-03-14T05:08:12.598954591Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 474 len 36864 data: 11 8 13 17 18 19 18 16 17 02/30 Read block 104 len 24576 03/30 Write block 559 len 24576 data: 12 13 9 8 8 8 04/30 Write block 418 len 32768 data: 18 16 15 18 18 20 19 17 05/30 Flush 06/30 Flush 07/30 Write block 210 len 40960 data: 14 15 16 21 17 18 19 18 13 14 08/30 Read block 268 len 32768 09/30 Read block 492 len 24576 10/30 Read block 557 len 32768 11/30 Write block 370 len 40960 data: 22 21 17 14 14 15 15 12 15 15 12/30 Read block 510 len 16384 13/30 Read block 376 len 12288 14/30 Read block 114 len 36864 15/30 Read block 124 len 40960 16/30 Read block 363 len 12288 17/30 Read block 481 len 32768 18/30 Write block 54 len 32768 data: 16 14 10 10 10 12 14 12 19/30 Flush 20/30 Write block 96 len 12288 data: 14 13 8 21/30 Read block 179 len 12288 22/30 Read block 328 len 28672 23/30 Write block 516 len 24576 data: 14 15 14 16 15 15 24/30 Read block 48 len 24576 25/30 Read block 441 len 12288 26/30 Flush 27/30 Write block 90 len 20480 data: 13 15 13 12 9 28/30 Write block 338 len 4096 data: 19 29/30 Write block 367 len 12288 data: 22 23 23 30/30 Write block 361 len 36864 data: 14 15 11 11 12 17 23 24 24 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:24 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1006 Acked Write 10 Done Done Done false 1007 Acked Read 8 Done Done Done false 1008 Acked Read 6 Done Done Done false 1009 Acked Read 8 Done Done Done false 1010 Acked Write 10 Done Done Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 3 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 3 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 6 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 19 2 23 Done 22 5 22 49 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1005)) Some(JobId(1025)) Downstairs last five completed: 1005 1004 1003 1002 1001 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-03-14T05:08:12.61241817Z","hostname":"w-01KKNBC4JFE5614WS1F0FV6QHH","pid":1997} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 196 196 196 365 365 365 F F F 2 040-059 198 196 198 369 365 369 F F F <--- 3 060-079 198 196 198 369 365 369 F F F <--- 4 080-099 198 194 198 370 363 370 T F T <--- 5 100-119 196 196 196 365 365 365 F F F 6 120-139 194 194 194 363 363 363 F F F 7 140-159 196 196 196 365 365 365 F F F 8 160-179 194 194 194 363 363 363 F F F 9 180-199 194 194 194 363 363 363 F F F 10 200-219 198 193 198 369 362 369 F T F <--- 11 220-239 194 194 194 363 363 363 F F F 12 240-259 196 196 196 365 365 365 F F F 13 260-279 188 188 188 354 354 354 F F F 14 280-299 196 196 196 365 365 365 F F F 15 300-319 197 197 197 366 366 366 F F F 16 320-339 197 197 197 366 366 366 T F T <--- 17 340-359 191 191 191 360 360 360 F F F 18 360-379 198 193 198 369 362 369 T T T <--- 19 380-399 196 196 196 365 365 365 F F F 20 400-419 198 198 198 367 367 367 F F F 21 420-439 198 198 198 367 367 367 F F F 22 440-459 193 193 193 362 362 362 F F F 23 460-479 198 198 198 367 367 367 F F F 24 480-499 198 198 198 367 367 367 F F F 25 500-519 198 193 198 370 362 370 F F F <--- 26 520-539 198 197 198 370 366 370 F F F <--- 27 540-559 198 198 198 367 367 367 F F F 28 560-579 198 198 198 367 367 367 F F F 29 580-599 193 193 193 362 362 362 F F F Max gen: 198, Max flush: 370 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 342 342 342 F F F 1 020-039 196 196 196 365 365 365 F F F 2 040-059 198 198 198 369 369 369 F F F 3 060-079 198 198 198 369 369 369 F F F 4 080-099 199 199 199 371 371 371 F F F 5 100-119 196 196 196 365 365 365 F F F 6 120-139 194 194 194 363 363 363 F F F 7 140-159 196 196 196 365 365 365 F F F 8 160-179 194 194 194 363 363 363 F F F 9 180-199 194 194 194 363 363 363 F F F 10 200-219 198 198 198 369 369 369 F F F 11 220-239 194 194 194 363 363 363 F F F 12 240-259 196 196 196 365 365 365 F F F 13 260-279 188 188 188 354 354 354 F F F 14 280-299 196 196 196 365 365 365 F F F 15 300-319 197 197 197 366 366 366 F F F 16 320-339 199 199 199 371 371 371 F F F 17 340-359 191 191 191 360 360 360 F F F 18 360-379 199 199 199 371 371 371 F F F 19 380-399 196 196 196 365 365 365 F F F 20 400-419 198 198 198 367 367 367 F F F 21 420-439 198 198 198 367 367 367 F F F 22 440-459 193 193 193 362 362 362 F F F 23 460-479 198 198 198 367 367 367 F F F 24 480-499 198 198 198 367 367 367 F F F 25 500-519 198 198 198 370 370 370 F F F 26 520-539 198 198 198 370 370 370 F F F 27 540-559 198 198 198 367 367 367 F F F 28 560-579 198 198 198 367 367 367 F F F 29 580-599 193 193 193 362 362 362 F F F Max gen: 199, Max flush: 371 7:36 Test duration Test completed real 7:35.930449080 user 36.146153074 sys 17.275829053 trap 0.015044693 tflt 0.072756674 dflt 0.068652643 kflt 0.003606011 lock 12:12:44.063628030 slp 37:11.709718176 lat 1.364795192 stop 0.025616160